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Abstract 


Defence  Research  and  Development  Canada  (DRDC)  Toronto  and  the  Canadian  Defence 
Academy  (CDA)  have  embarked  upon  a  collaborative  research  effort  to  identify  the  most 
promising  topics  for  Research  and  Development  (R&D)  on  accelerating  learning  and  maximizing 
knowledge/skill  retention  that  could  yield  the  greatest  benefit  to  the  Canadian  Forces  (CF)  within 
five  years.  This  report  examines  the  two  deliverables  of  this  project,  a  literature  review  and  a 
workshop. 

The  literature  generated  a  list  of  potential  research  concepts  related  to  accelerated  learning  and 
retention.  A  review  of  26  articles  from  the  relevant  literature  showed  a  range  of  factors  likely  to 
impact  on  learning  and  retention.  These  included  characteristics  of  the  student  (e.g.,  cognitive 
ability  and  motivation),  the  instructor,  delivery  media  (e.g.,  classroom-based  instruction  vs. 
computer-based  instruction),  delivery  methods  (e.g.,  scheduling,  feedback  and  testing)  and  issues 
related  to  retention.  It  was  noted  that  the  factors  that  influence  skill  acquisition  are  not  necessarily 
the  same  ones  that  influence  skill  retention.  The  literature  review  examined  the  available  research 
exploring  the  impact  of  these  factors  on  learning  and  retention,  and  identified  gaps  and  emerging 
research  questions. 

Following  the  identification  of  these  15  research  topics,  a  workshop  (coordinated  and  facilitated 
by  the  CDA)  was  convened  in  Kingston,  Ontario  in  order  to  explore  the  merit  and  importance  of 
these  topics  from  the  perspective  of  subject  matter  experts,  in  order  to  guide  future  research  and 
development  efforts.  Thirteen  stakeholders,  representing  Navy,  Army  and  Air  force,  discussed 
and  provided  ratings  on  each  of  the  1 5  topics  on  each  of  1 1  evaluation  criteria  such  as  the 
scientific  merit  of  the  topic  area,  its  alignment  with  CF/DND  goals,  and  the  potential  for  national 
or  international  collaboration  on  the  topic. 

Quantitative  results  showed  that  simulator-based  instruction  was  rated  as  being  the  most 
promising  area  for  future  research,  followed  by  instructor  attributes,  collaborative  learning, 
distance  learning  and  computer-based  instruction.  Participants  also  provided  rich  insights  about 
each  of  the  15  topic  areas,  and  raised  potential  research  questions  that  could  be  addressed  in 
future  research  and  development  efforts. 
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Resume 


Recherche  et  developpement  pour  la  defense  Canada  (RDDC)  -  Toronto  et  l’Academie 
canadienne  de  la  Defense  (ACD)  collaborent  a  un  projet  de  recherche  sur  les  sujets  de  recherche 
et  developpement  les  plus  prometteurs  concemant  les  fatjons  d’accelerer  l’apprentissage  et  de 
maximiser  la  conservation  des  connaissances  et  des  competences  acquises  qui  pourraient  etre  les 
plus  avantageuses  pour  les  Forces  canadiennes  (FC)  au  cours  des  cinq  prochaines  annees.  Le 
present  rapport  fait  le  point  sur  deux  realisations  du  projet :  une  analyse  de  la  documentation 
parue  sur  le  sujet  et  un  atelier. 

L’analyse  de  la  documentation  a  permis  d’extraire  une  liste  de  concepts  de  recherche  potentiels 
lies  a  l’apprentissage  accelere  et  a  la  conservation  des  connaissances.  L’analyse  de  26  articles  a 
mis  au  jour  une  gamme  de  facteurs  susceptibles  d’influer  sur  l’apprentissage  et  la  conservation 
des  connaissances  acquises.  11  s’agit  notamment  des  caracteristiques  de  la  personne  qui  apprend 
(p.  ex.  :  sa  capacite  cognitive  et  sa  motivation),  de  l’instructeur,  des  moyens  utilises  pour  donner 
la  formation  (p.  ex.  :  enseignement  donne  en  classe  ou  enseignement  informatise),  des  methodes 
d’enseignement  utilisees  (p.  ex.  :  etablissement  du  calendrier  de  formation,  retroaction  et 
examens)  et  des  questions  relatives  a  la  conservation  des  connaissances  acquises.  On  a  fait 
observer  que  les  facteurs  qui  influent  sur  l’acquisition  des  competences  ne  sont  pas 
necessairement  les  memes  que  ceux  qui  influent  sur  la  conservation  des  connaissances  et  des 
competences  acquises.  Lorsqu’on  a  analyse  la  documentation  sur  le  sujet,  on  s’est  penche  sur 
l’incidence  de  ces  facteurs  sur  l’apprentissage  et  la  conservation  des  connaissances  et  on  a  releve 
des  lacunes  ainsi  que  des  nouvelles  questions  sur  la  recherche. 

Apres  avoir  recense  quinze  sujets  de  recherche,  on  a  organise  un  atelier  (coordonne  et  anime  par 
l’ACD)  a  Kingston  au  cours  duquel  on  a  explore  les  merites  et  l’importance  de  ces  questions  du 
point  de  vue  des  specialistes  pour  guider  les  travaux  futurs  de  recherche  et  de  developpement. 
Treize  intervenants,  representant  la  Marine,  l’Armee  de  terre  et  la  Force  aerienne,  ont  participe 
aux  discussions  et  cote  chacun  des  quinze  sujets  en  regard  de  chacun  des  onze  criteres 
d’ evaluation,  comnie  le  merite  scientifique  de  la  discipline  concemee,  sa  pertinence  par  rapport 
aux  objectifs  des  FC  ou  du  MDN  et  le  potentiel  de  collaboration  au  niveau  national  ou 
international. 

Les  resultats  quantitatifs  obtenus  montrent  que  1’  instruction  par  simulateur  est  consideree  comnie 
le  secteur  le  plus  prometteur  pour  la  recherche  future,  suivie  des  caracteristiques  des  instructeurs, 
l’apprentissage  en  collaboration,  l’apprentissage  a  distance  et  l’instruction  par  ordinateur.  Les 
participants  ont  egalement  donne  des  points  de  vue  tres  eclaires  sur  chacun  des  quinze  sujets  et 
fait  part  de  questions  de  recherche  potentielles  pour  les  futurs  travaux  de  recherche  et  de 
developpement. 
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Executive  summary 


Accelerated  Learning  and  Retention:  Literature  Review  and 
Workshop  Review: 

Adams,  B.D.;  Karthaus,  C.;  Rehak,  L.A.;  DRDC  Toronto  CR  2011-105;  Defence 

R&D  Canada  -  Toronto;  March  201 1 . 

Introduction  or  background:  Defence  Research  and  Development  Canada  (DRDC)  Toronto 
and  the  Canadian  Defence  Academy  (CDA)  have  embarked  upon  a  collaborative  research  effort 
to  identify  the  most  promising  topics  for  Research  and  Development  (R&D)  on  accelerating 
learning  and  maximizing  knowledge/skill  retention  that  could  yield  the  greatest  benefit  to  the 
Canadian  Forces  (CF)  within  five  years.  This  report  examines  the  two  deliverables  of  this  project, 
a  literature  review  and  a  workshop. 

The  goal  of  the  literature  review  was  to  generate  a  list  of  potential  research  concepts  related  to 
accelerated  learning  and  retention.  This  review  required  examining  26  articles  from  the  relevant 
scientific,  military  and  related  literature  relevant  to  the  question  of  accelerated  learning  and 
retention.  Although  the  literature  related  to  accelerated  learning  was  somewhat  underdeveloped, 
the  review  showed  a  range  of  factors  likely  to  impact  on  learning  and  retention.  These  included 
characteristics  of  the  student  (e.g.,  cognitive  ability  and  motivation),  the  instructor,  delivery 
media  (e.g.,  classroom-based  instruction  vs.  computer-based  instruction),  delivery  methods  (e.g., 
scheduling,  feedback  and  testing)  and  issues  related  to  retention;  noting  that  the  factors  that 
influence  skill  acquisition  are  not  necessarily  the  same  ones  that  influence  skill  retention.  The 
literature  review  examined  the  available  research  exploring  the  impact  of  these  factors  on  learning 
and  retention,  and  identified  gaps  and  emerging  research  questions. 

Following  the  identification  of  these  15  research  topics,  a  workshop  (coordinated  and  facilitated 
by  the  CDA)  was  convened  in  Kingston  in  order  to  explore  the  merit  and  importance  of  these 
topics  from  the  perspective  of  subject  matter  experts,  in  order  to  guide  future  research  and 
development  efforts.  Workshop  participants  were  provided  with  definitions  of  the  topic  areas  to 
ensure  common  understanding,  and  they  assembled  into  break-out  groups  facilitated  by  CDA 
personnel.  A  total  of  13  stakeholders,  representing  Navy,  Army  and  Air  force,  discussed  and 
provided  ratings  on  each  of  the  1 5  topics  on  each  of  1 1  evaluation  criteria.  These  criteria  included 
the  scientific  merit  of  the  topic  area,  its  alignment  with  CF/DND  goals,  and  the  potential  for 
national  or  international  collaboration  on  the  topic.  Participants  rated  the  importance  of  each  topic 
on  each  criterion  using  a  rating  scale. 

Results:  The  results  of  the  workshop  were  analyzed  from  both  qualitative  and  quantitative 
perspectives.  Qualitative  remarks  made  by  the  participants  provided  rich  insights  about  each  of 
the  15  topic  areas,  in  terms  of  how  they  are  defined  and  the  issues  within  each  area  that  are 
relevant  to  their  own  organizations.  They  also  raised  potential  research  questions  that  could  be 
addressed  in  future  research  and  development  efforts.  Quantitative  analyses  of  the  importance 
ratings  showed  that  simulator-based  instruction  was  rated  as  being  the  most  promising  area  for 
future  research,  followed  by  instructor  attributes,  collaborative  learning,  distance  learning  and 
computer-based  instruction.  Flowever,  importance  ratings  for  all  topics  were  very  closely  spaced. 
Moreover,  even  the  lowest  rated  topic,  namely  scheduling,  was  rated  as  being  above  the  midpoint 
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of  the  rating  scale,  suggesting  that  participants  saw  all  15  topics  as  relevant  and  worthy  of 
research  attention. 

Future  plans:  This  information  will  be  used  to  guide  future  research  and  development  efforts. 
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pour  la  defense  Canada  -  Toronto;  Mars  2011. 

Introduction  ou  contexte  :  Recherche  et  developpement  pour  la  defense  Canada  (RDDC)  - 
Toronto  et  l’Academie  canadienne  de  la  Defense  (ACD)  collaborent  a  un  projet  de  recherche  sur 
les  sujets  de  recherche  et  developpement  les  plus  prometteurs  concemant  les  fagons  d’accelerer 
l’apprentissage  et  de  maximiser  la  conservation  des  connaissances  et  des  competences  acquises 
qui  pourraient  etre  les  plus  avantageuses  pour  les  Forces  canadiennes  (FC)  au  cours  des  cinq 
prochaines  annees.  Le  present  rapport  fait  le  point  sur  deux  realisations  du  projet :  une  analyse  de 
la  documentation  parue  sur  le  sujet  et  un  atelier. 

L’analyse  de  la  documentation  a  pemiis  d’extraire  une  liste  de  concepts  de  recherche  potentiels 
lies  a  l’apprentissage  accelere  et  a  la  conservation  des  connaissances.  L’analyse  de  26  articles  a 
mis  au  jour  une  gamrne  de  facteurs  susceptibles  d’influer  sur  l’apprentissage  et  la  conservation 
des  connaissances  acquises.  11  s’agit  notamment  des  caracteristiques  de  la  personne  qui  apprend 
(p.  ex.  :  sa  capacite  cognitive  et  sa  motivation),  de  l’instructeur,  des  moyens  utilises  pour  donner 
la  formation  (p.  ex.  :  enseignement  donne  en  classe  ou  enseignement  informatise),  des  methodes 
d’enseignement  utilisees  (p.  ex.  :  etablissement  du  calendrier  de  formation,  retroaction  et 
examens)  et  des  questions  relatives  a  la  conservation  des  connaissances  acquises.  On  a  fait 
observer  que  les  facteurs  qui  influent  sur  l’acquisition  des  competences  ne  sont  pas 
necessairement  les  memes  que  ceux  qui  influent  sur  la  conservation  des  connaissances  et  des 
competences  acquises.  Lorsqu’on  a  analyse  la  documentation  sur  le  sujet,  on  s’est  penche  sur 
l’incidence  de  ces  facteurs  sur  l’apprentissage  et  la  conservation  des  connaissances  et  on  a  releve 
des  lacunes  ainsi  que  des  nouvelles  questions  sur  la  recherche. 

Apres  avoir  recense  quinze  sujets  de  recherche,  on  a  organise  un  atelier  (coordonne  et  anime  par 
l’ACD)  a  Kingston  au  cours  duquel  on  a  explore  les  merites  et  l’importance  de  ces  questions  du 
point  de  vue  des  specialistes  pour  guider  les  travaux  futurs  de  recherche  et  de  developpement.  Les 
participants  de  1’ atelier  ont  regu  des  definitions  et  des  sujets  faisant  en  sorte  que  tous  aient  une 
comprehension  commune  et  ont  ete  repartis  en  petits  groupes  animes  par  le  personnel  de  l’ACD. 
Treize  intervenants,  representant  la  Marine,  l’Armee  de  terre  et  la  Force  aerienne,  ont  participe 
aux  discussions  et  cote  chacun  des  quinze  sujets  en  regard  de  chacun  des  onze  criteres 
d’ evaluation,  cornme  le  merite  scientifique  de  la  discipline  concemee,  sa  pertinence  par  rapport 
aux  objectifs  des  FC  ou  du  MDN  et  le  potentiel  de  collaboration  au  niveau  national  ou 
international.  Les  participants  ont  cote  l’importance  de  chaque  sujet  en  fonction  de  chaque  critere 
en  utilisant  une  echelle  de  cotation. 

Resultats  :  Les  resultats  de  l’atelier  ont  ete  analyses  a  la  fois  du  point  de  vue  qualitatif  et  du  point 
de  vue  quantitatif.  Les  participants  ont  formule  des  observations  qualitatives  tres  instructives  sur 
chacun  des  quinze  sujets  traites,  tant  sur  le  plan  de  la  fagon  dont  ils  defmissent  ces  sujets  que  sur 
celui  des  questions  a  l’interieur  de  chaque  domaine  qui  sont  pertinentes  pour  leurs  organisations 
respectives.  11s  ont  egalement  fait  part  de  questions  de  recherche  potentielles  pour  les  futurs 
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travaux  de  recherche  et  developpement.  Les  resultats  quantitatifs  obtenus  sur  les  cotes  attribuees  a 
1’importance  des  sujets  montrent  que  l’instruction  par  simulateur  est  consideree  comnie  le  secteur 
le  plus  prometteur  pour  la  recherche  future,  suivie  des  caracteristiques  des  instructeurs,  de 
l’apprentissage  en  collaboration,  de  l’apprentissage  a  distance  et  de  l’instruction  par  ordinateur. 
Toutefois,  les  cotes  relatives  a  I' importance  de  tous  les  sujets  etaient  tres  serrees.  De  plus,  meme 
le  sujet  qui  avait  regu  la  cote  la  plus  basse,  soit  l’etablissement  du  calendrier  de  la  formation,  avait 
une  cote  superieure  au  point  central  de  l’echelle  de  cotation,  ce  qui  donne  a  penser  que  les 
participants  consideraient  que  les  quinze  sujets  etaient  pertinents  et  dignes  de  faire  l’objet  de 
travaux  de  recherche. 

Perspectives  :  Ces  renseignements  serviront  aux  projets  tuturs  en  matiere  de  recherche  et  de 
developpement. 
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Introduction 


1.1  Background 

Defence  Research  and  Development  Canada  (DRDC)  Toronto  and  the  Canadian  Defence 
Academy  (CDA)  have  embarked  upon  a  collaborative  research  effort  to  identify  the  most 
promising  topics  for  Research  and  Development  (R&D)  on  Accelerating  Learning  and 
maximizing  knowledge/skill  retention  that  could  yield  the  greatest  benefit  to  the  Canadian  Forces 
(CF)  within  five  years.  HSI®  has  undertaken  this  scoping  study  to  support  DRDC  Toronto  and 
CDA  in  achieving  this  objective.  Follow-on  R&D  activities  (including  possible  Applied  Research 
Projects  or  ARPs)  will  be  generated  from  the  findings  of  the  scoping  study. 

Accelerated  learning  is  defined  as  occurring  when  there  is  “more  learned  in  fixed  time  (resources) 
or  less  time  (resources)  to  reach  fixed  level  of  learning”  (Statement  of  Work,  2010).  In  other 
words,  accelerated  learning  allows  a  shortened  timeframe  for  learning  with  a  comparative  amount 
of  material  learned  and  retained.  Accelerated  learning  is  of  particular  importance  to  the  military 
because  with  appropriate  development  and  integration  training  can  aid  personnel  to  “quickly  and 
effectively  gain  the  competence  required  for  their  deployment  or  to  retain  the  skills  they  have 
mastered...”  (Andrews  &  Fitzgerald,  2010,  p.4). 

1.2  Scope  and  Objectives 

This  report  represents  the  deliverable  of  this  project  and  details  the  methods  and  findings  of  a 
literature  review  and  workshop  on  accelerated  learning  and  retention.  The  goal  of  the  literature 
review  was  to  generate  a  comprehensive  list  of  potential  research  concepts  related  to  accelerated 
learning  and  retention  that  DRDC  Toronto  and  CDA  may  want  to  pursue  in  future  R&D  efforts. 
These  concepts  served  as  the  basis  for  workshop  discussions.  Specifically,  the  literature  review 
served  to:  (1)  identify  the  physical,  psychological  and  social  factors  that  could  positively  or 
negatively  affect  learning  rate  and  retention;  and  (2)  provide  preliminary  recommendations  for 
R&D  that  could  yield  the  best  value  for  the  CF.  In  the  workshop,  the  list  of  potential  research 
topics  identified  through  the  literature  review  were  examined  by  experts  in  the  field  and  rated 
according  to  established  criteria,  culminating  in  a  short  list  of  topics  available  for  further  research. 

1.3  Outline 

The  report  is  divided  into  4  main  sections.  The  next  section  outlines  the  methods  and  findings  of  a 
literature  review  while  Section  3  details  the  workshop  methods  and  results.  The  final  section 
conveys  overall  conclusions. 


DRDC  Toronto  CR  201 1-105 


1 


2  Literature  Review 


2.1  Method 

2.1.1  General  Approach 

For  this  literature  review,  we  used  what  can  be  described  as  a  “funnelling”  approach.  We 
identified  many  concepts  and  then  selected  the  most  promising  concepts  to  recommend  for  future 
R&D.  As  such,  the  project  began  by  broadly  exploring  trends,  issues  and  factors  that  have  shown 
promise  to  improve  education,  learning,  training  and  retention.  To  begin,  the  project  team 
identified  key  words  pertaining  to  accelerated  learning  and  retention  to  be  used  in  the  literature 
search.  Selected  databases  were  then  searched  using  those  terms.  Articles  were  gathered  based  on 
their  relevance.  From  this  list  of  articles,  26  were  selected  and  reviewed.  The  literature  was  then 
used  to  generate  a  list  of  potential  research  topics. 

This  section  details  the  initial  stages  of  this  project  leading  up  to  the  identification  of  concepts 
and  topics  within  the  accelerated  learning  and  retention  literature.  The  searches  were  geared  to 
include  all  different  types  of  skills.  As  a  result,  the  body  of  literature  examined  included  more 
complex  cognitive  skills  as  well  as  procedural  skills  and  motor  skills.  The  approach  was  to 
examine  the  vast  literature  broadly  looking  at  both  academic  and  public  sources.  The  following 
sections  outline  the  specific  keywords,  databases,  selection  of  articles,  articles  reviewed  and  a 
summary  of  results. 

2.1.2  Keywords 

To  begin,  a  keyword  list  was  generated.  This  process  involved  a  brainstorming  session  with  all 
members  of  the  research  team,  and  relied  on  their  cumulative  knowledge  and  experience  with  the 
pertinent  scientific  and  military  domains.  Feedback  was  provided  by  the  scientific  authority  (SA) 
and  technical  authority  (TA)  and  a  final  list  of  keywords  was  developed  to  focus  the  literature 
search.  The  team  established  a  number  of  core  concepts,  which  included  accelerated  learning, 
skill  retention,  and  knowledge  retention  (Table  1) 


Table  1  Keywords 


Core  Concept 

Primary  Keywords 

Secondary  Keywords 

Accelerated  learning 

Learn* 

Acqui*,  knowledge,  study 

Accelerate* 

Speed,  rate,  quick,  fast*,  expedit* 

Skill/Knowledge 

Retention 

Retention 

Fading,  retain,  re-train,  refresher,  recogni* 

Training 

Transfer  of,  transf*  education,  learning  application, 
teach*,  instruct*,  simulation,  experiential 

Challenges 

Obstacles,  ineffective*,  problems,  opportunit* 

Improve 

Novel,  promising,  trend*,  improve*,  performance, 
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enhance,  next  generation,  effective*,  advance*,  potential, 
better 

Domain 

Military,  NASA,  medical,  higher  educ*,  post-secondary, 
corporate,  emergency,  police,  fire  fighting,  search  and 
rescue 

In  a  teleconference  held  with  the  HSIs  team  and  the  project  TA,  it  was  decided  that  all  core 
concepts  would  be  searched  in  relevant  databases.  In  order  to  narrow  the  search  when  required, 
core  concepts  and/or  primary  key  words  were  used.  Secondary  keywords  were  intended  to 
provide  further  narrowing  if  necessary,  however,  they  were  not  always  required. 

2.1.3  Databases 

The  keywords  were  used  to  search  a  number  of  databases  (Table  2)  with  the  goal  of  identifying 
review  articles  and  empirical  studies  in  high  quality  journals.  Second,  other  media  sources  (e.g. 
The  New  York  Times,  Maclean’s,  educational  organizations,  and  additional  sources  identified  in 
the  preliminary  research  stage)  were  searched  using  the  keywords  identified.  Finally,  RSS  (Really 
Simple  Syndication)  feeds  for  search  results  using  different  combinations  of  these  keywords  were 
set  up  and  reviewed  daily  by  the  project  team  to  further  identify  relevant  sources  and  key  search 
terms. 
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Table  2  Databases  Searched 


Database  /  Resource 

Annual  Review  of  Psychology 

Psychology  Bulletin  (included  under  APA  search) 

APA  (reviews  only;  journal  articles) 

Scientific  America 

Maclean's 

Accelerated  Learning  workshop  (D.  Andrews) 

NRC  (National  Research  Council) 

PsychINFO  -  included  in  APA 

NTIS  /  CISTI  (National  Technical  Information  Service/  Canadian  Institute  for  Scientific 
and  Technical  Information) 

STINET  (Scientific  and  Technical  Information  Network) 

Google  Scholar 

TED  Talks  (Technology,  Entertainment,  Design) 

Journal  of  Educational  Computing  Research 

Contemporary  Educational  Psychology 

Metacognition  and  Learning 

Handbook  of  Educational  Psychology  (searched  within  2nd  Ed.  2007) 

New  York  Times :  Education 

ARI  (Army  Research  Institute) 

The  Economist 

West  Point 

Harvard  educational  publishing  group 

Conferences 

TEKL  (Technologically  externalized  knowledge  and  learning) 

Others 

Psychological  Review  (included  in  APA) 

Journal  of  Educational  Psychology  (included  in  APA) 

Defence  Research  Reports  Database 

HSI  technical  and  contract  reports 
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2.1.4  Review  of  Articles 


After  the  article  selection  process  was  complete,  an  initial  set  of  articles  was  reviewed  by  the 
research  team  at  HSI®. 

2.1.5  Limitations  of  the  Review 

The  primary  limitation  of  the  review  is  that  it  covers  a  very  limited  number  of  articles,  relative  to 
the  breadth  of  the  problem  space.  As  will  be  discussed  in  the  review,  the  relevant  literature  is  also 
relatively  underdeveloped  in  some  areas,  particularly  with  respect  to  accelerated  learning. 

2.2  Introduction  to  Accelerated  Learning 

This  chapter  explores  the  concept  of  accelerated  learning,  in  terms  of  its  definition,  and  the 
alleged  effects.  The  sections  that  follow  address  the  retention  of  accelerated  learning,  and 
research  themes  and  questions  relevant  to  the  concept  of  accelerated  learning. 

2.2.1  Definition 

This  review  focuses  on  literature  relevant  to  accelerated  learning,  defined  as  “more  learned  in 
fixed  time  (resources)  or  less  time  (resources)  to  reach  fixed  level  of  learning”  (Statement  of 
Work,  2010).  In  other  words,  accelerated  learning  requires  the  typical  timeframe  for  learning  to 
be  shortened  (e.g.,  4  to  10  weeks  versus  16  to  20  weeks  in  a  traditional  college  course), 
participating  in  fewer  learning  hours  (e.g.,  20-32  hours  versus  a  traditional  48-60  hours  in  a 
typical  college  course),  with  a  comparative  amount  of  material  learned  and  retained.  This  can 
sometimes  be  contused  with  compressed  courses,  typically  found  in  distance  education,  where 
learning  is  intensified.  Students  are  required  to  spend  the  same  amount  of  time  learning  (40-45 
hours)  in  a  shorter  amount  of  frame  (typically  6-8  weeks)  (see  Capuzzi  Simon,  2007). 

Andrews  and  Fitzgerald  (2010,  p.  26)  describe  the  result  of  accelerated  learning  as  “the 
attainment  of  competence”.  This  definition  clarifies  that  the  goal  of  accelerated  learning  is  simply 
to  promote  higher  levels  of  competence  in  whatever  skill  area  is  required.  However,  it  should  be 
noted  that  they  focus  on  accelerated  learning  from  the  perspective  of  complex  rather  than 
mundane  skills. 

The  goals  of  accelerated  learning  are  explicit  in  work  by  Andrews  and  Fitzgerald  (2010,  p.  20). 
Specifically,  these  goals  are: 

♦  Speed  knowledge  acquisition  and  increase  duration  and  quantity  of  retention 

♦  Cultivate  deeper  learning  and  expertise 

♦  Acquisition  of  a  robust  knowledge/skill  base 

♦  Generalizability  to  similar  tasks 

♦  Retention  of  complex  skills 
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Andrews  and  Fitzgerald  (2010,  p.  21)  also  present  a  number  of  core  principles  that  they  argue 
underlie  the  current  understanding  of  accelerated  learning.  They  argue  that  simple  routine  practice 
is  not  enough  to  promote  accelerated  learning,  and  they  offer  the  following  guidelines: 

♦  Skills  must  be  constantly  stretched,  with  increasing  challenges 

♦  High  internal  motivation  is  required  to  promote  persistence  on  hard  tasks 

♦  Rich,  meaningful  feedback  is  an  important  aspect  of  practice, 

♦  Mentoring  is  a  critical  aspect  of  practice 

♦  Providing  for  the  unique  learning  styles  of  individuals  through  individualized  or 
tailored  practice  is  necessary 

2.2.2  Effects  of  Accelerated  Learning 

The  concept  of  accelerated  learning  has  been  a  “hot  topic”  in  learning  since  1991  (Erickson, 
1991).  Accelerated  learning  can  be  defined  as  any  instance  in  which  the  outcome  is  a  level  of 
learning  or  proficiency  that  has  occurred  in  less  time  than  traditionally  or  previously  required. 
Such  resource  savings  have  made  accelerated  learning  an  attractive  concept  to  industry, 
academia,  individuals,  and  the  military.  Well-known  companies,  such  as  Shell  Oil,  General 
Motors  (Erickson,  1991),  American  Airlines,  and  Kodak  (Meier,  2000)  have  used  accelerated 
learning  in  training  programs.  Table  3  provides  concrete  examples  of  some  of  the  results  offered 
by  approaches  intended  to  promote  accelerated  learning. 


Table  3  Examples  of  accelerated  learning  outcomes  in  organizations  (Meier,  2000,  p.  xxix) 


Company 

Application 

Results 

American  Airlines 

Reservationists  Training 

Reduced  training  time  for  a  lesson  by  50%.  Improved 
the  retention  significantly. 

Verizon 

Customer  Service  Rep 

Cut  training  time  in  half  while  improving  measurable 
performance. 

Chevron 

Fire  Extinguisher 

Training 

Reduced  training  time  by  50%  while  achieving  same  or 
better  learning. 

Consolidated  Edison 

Cable  Splicing  Course 

Passing  rate  increased  from  30%  to  100%  in  same 
time. 

Commonwealth 

Edison 

Time  Keeper  Training 

Cut  class  time  in  half  while  greatly  improving  test 
scores,  long-term  retention  and  student  evaluations. 

Florida  Community 

Lotus  1-2-3  Course 

Students  learned  75%  faster  while  enjoying  the  training 
much  more. 

Fortune  100  Midwest 
Manufacturer 

Inventory  Management 
Course 

Reduced  training  time  by  60%  while  improving 
learning. 

Kodak 

Electronics  Course 

Cut  training  time  by  a  third  and  improved  long-term 
retention  by  25%. 

Major  U.S. 
Semiconductor 

Flazcom  and  Safety 
Training 

Improved  measurable  learning  by  5x  in  the  same  time 
frame. 

Verizon 

Telephone  Skills 

Training 

Cut  training  time  by  50%  and  doubled  the  learning. 

Travellers  Insurance 

Medical  Claim  Benefits 
Training 

Cut  training  time  by  20%  improving  test  scores  by 
almost  5x. 
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Company 

Application 

Results 

Major  Retail  Chain 

Coaching  Skills  for 
Managers 

Reduced  training  time  by  75%  while  achieving  better 
results. 

In  conclusion,  a  number  of  positive  accelerated  learning  outcomes  have  been  noted  in  the 
literature.  However,  it  is  important  to  explore  the  factors  that  might  underlie  these  outcomes. 

The  following  sections  explore  specific  sets  of  factors  noted  within  the  relevant  learning  and 
retention  literature  as  contributing  to  positive  learning  outcomes.  These  include  student  attributes, 
instructor  attributes,  delivery  media,  delivery  methods,  and  skills  and  knowledge  retention. 

2.3  Student  Attributes 

This  section  considers  the  available  literature  relevant  to  attributes  of  the  student  that  have  been 
argued  to  impact  on  learning  and  (to  the  extent  possible)  accelerated  learning. 

2.3.1  Cognitive  Ability 

Cognitive  ability  is  often  identified  in  the  literature  as  increasing  the  rate  at  which  information  is 
acquired.  Cognitive  ability  is  typically  defined  as  general  intelligence  (or  g)  (e.g.,  Salas  & 
Cannon-Bowers,  2001). 

There  is  very  good  evidence  in  the  literature  that  individuals  with  better  cognitive  ability  tend  to 
learn  at  a  faster  rate,  thereby  achieving  a  higher  overall  level  of  learning  within  the  same  training 
duration.  A  consistent  pattern  of  results  about  the  power  of  cognitive  ability  is  aptly  summed  by 
Salas  and  Cannon-Bowers  (2001,  p.  478),  who  note  “Clearly,  those  who  have  high  cognitive 
ability  (all  other  things  being  equal)  will  likely  learn  more  and  succeed  in  training”.  Research  by 
Blume,  Ford,  Baldwin,  and  Huang  (2010)  suggests  that  the  ability  to  learn  better  as  the  result  of 
cognitive  ability  may  carry  over  to  being  able  to  actually  transfer  work  skills.  For  example,  based 
on  their  meta-analysis  of  the  transfer  of  training  literature,  Blume  et  al.  (2010)  found  that  the 
student  characteristic  most  strongly  related  to  training  transfer  was  cognitive  ability,  and  it 
showed  a  moderate  (r  =  .37)  correlation  with  transfer.  However,  an  important  gap  identified  by 
Salas  and  Cannon-Bowers  (2001)  is  research  exploring  students  with  lower  cognitive  ability,  and 
how  learning  can  be  optimized  for  them.  They  also  argue  that  understanding  the  interaction 
between  cognitive  aptitude  and  the  nature  of  specific  jobs  or  tasks  is  another  area  also  in  need  of 
further  attention. 

Two  other  specific  forms  of  cognitive  ability  are  evident  in  the  literature.  Hoffman,  Feltovich, 
Fiore,  Klein,  and  Ziebell  (2009)  note  the  importance  of  cognitive  flexibility  and  cognitive 
transformation.  When  confronted  with  complex  decisions,  learners  often  use  simplistic 
constructions  of  the  problem  that  do  not  capture  the  complexity  of  counterarguments  (or 
alternative  ways  of  thinking).  These  constructions  are  called  knowledge  shields.  Cognitive 
flexibility  is  defined  as  being  “willing  and  able  to  recognize  and  overcome  the  knowledge 
shields”  (Hoffman  et  al.,  2009,  p.  19).  Cognitive  transformation,  they  argue,  is  a  closely  related 
concept,  that  emphasizes  the  importance  of  unlearning.  Like  cognitive  flexibility,  cognitive 
transformation  allows  people  to  move  beyond  simplistic  mental  models  to  achieve  deep 
understanding. 
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Although  it  has  different  nuances  in  the  available  literature,  cognitive  ability  has  a  strong  impact 
on  student  learning. 


2.3.2  Motivation 

Another  major  factor  related  to  the  effectiveness  of  accelerated  training  is  motivation.  A 
definition  of  training  motivation  in  the  literature  conceptualizes  it  as  “...the  direction,  effort, 
intensity  and  persistence  that  trainees  apply  to  learning-oriented  activities  before,  during  and  after 
training  (Kanfer,  1991,Tannenbaum  &  Yukl,  1992;  cited  in  Salas  &  Cannon-Bowers,  2001,  p. 
479). 

A  review  of  the  literature  by  Salas  and  Cannon-Bowers  (2001,  p.  479)  concluded  that  there  was 
very  good  evidence  of  the  power  of  motivation  to  learn  “on  their  skill  acquisition,  retention  and 
willingness  to  apply  the  newly  acquired  knowledge,  skill,  and  abilities  on  the  job”.  However,  they 
also  argue  that  strong  theoretical  frameworks  are  rare.  In  one  notable  example  of  such  a  model, 
Colquitt,  LePine,  and  Noe  (2000)  presented  and  tested  an  integrated  model  of  training  motivation. 
This  model  is  consistent  with  many  factors  identified  in  other  models  of  transfer  (e.g.,  Baldwin, 
Ford,  &  Blume,  2009).  These  include  student  attributes,  the  work  environment,  and  a  wide  range 
of  near  and  far  transfer  outputs  (e.g.,  declarative  knowledge  and  reactions,  and  job  performance). 
Results  of  their  analyses  showed  that  the  four  learning  outcomes  (declarative  knowledge,  skill 
acquisition,  post-training  self-efficacy,  reactions)  explained  about  half  of  the  variance  in  transfer. 
However,  when  more  distal  variables  (personality,  age,  situational  variables)  were  included,  a 
total  of  8 1  %  of  the  variance  in  transfer  could  be  explained.  These  results  support  the  assertion  that 
individual  attributes  along  with  relevant  situational  variables  do  have  a  direct  impact  on  the 
transfer  of  training  in  the  workplace,  over  and  above  what  students  take  away  from  the  training 
session  (as  measured  by  the  learning  outcomes).  This  research  shows  good  evidence  of  the 
importance  of  individual  attributes  such  as  motivation,  even  independently  of  training 
approaches. 

The  role  of  motivation  is  also  notable  in  the  literature  specifically  addressing  accelerated  learning. 
Motivating  students  to  learn  comes  directly  from  the  founding  principles  and  assumptions  of 
accelerated  learning.  Meier  (2000)  argues  that  accelerated  learning  requires  hard  work,  it  requires 
students  to  be  actively  involved  with  their  whole  mind  and  body,  and  learning  should  consist  of 
“creating”  rather  than  consumption.  High  levels  of  motivation  are  necessary  for  effective  learning 
and  lead  to  more  favourable  outcomes,  (see  Andrews  &  Fitzgerald,  2010;  Baldwin  et  al.,  2009; 
Cannon-Bowers  &  Bowers,  2008;  Meier,  2000;  Zipperer,  Klein,  Fitzgerald,  Kinnison  &  Graham, 
2003).  When  every  minute  counts  (as  is  the  case  in  accelerated  programs),  motivation  is  of 
critical  importance.  This  is  one  reason  why  training  interventions  aimed  to  increase  student 
motivation  such  as  gaming,  team  activities  (e.g.,  games,  brain  storming),  web-based  discussion, 
virtual  worlds  are  used  to  promote  learning  (see  Cannon-Bowers  &  Bowers,  2008;  Erickson, 
1991;  Meier,  2000).  Students  who  are  fully  engaged  and  motivated  to  learn  actually  acquire  more 
information  and  are  able  to  retain  it  longer  (see  Andrew  &  Fitzgerald,  2010;  Meier,  2000). 
Students  who  have  high  motivation  take  ownership  of  their  own  learning,  become  more  engaged, 
and  get  more  out  of  the  experience  (see  Cannon-Bowers  &  Bowers,  2008;  Meier,  2000). 
Exercises  such  as  goal  setting  and  encouraging  goal  commitment  increase  ownership  of  learning 
which  is  associated  with  effective  learning  (Schunk  &  Ertmer,  1999;  cited  in  Cannon-Bowers  & 
Bowers,  2008).  This  literature  suggests  that  highly  motivated  students  may  be  necessary  for 
accelerated  learning  to  occur. 


DRDC  Toronto  CR  201 1-105 


A  potentially  important  aspect  of  motivation  for  some  learners  is  the  need  to  see  and  understand 
the  relevance  of  training  efforts.  Skills  and  theory  can  help  military  members  understand  not  only 
what  tasks  need  to  be  done  and  how  they  need  to  be  done,  but  also  why  they  need  to  be  done. 
Understanding  the  relevance  of  training  can  also  help  learners  to  increase  motivation,  focus,  and 
recognize  when  and  how  specific  skills  should  be  applied  (see  Zipperer  et  al.,  2003). 

Whatever  the  focus,  then,  the  motivation  of  the  student  has  a  consistently  strong  impact  on 
learning. 

2.3.3  Learning  Styles 

Learning  styles  is  defined  in  the  literature  as  “...the  view  that  different  people  learn  information 
in  different  ways”  (Pashler,  McDaniel,  Rohrer,  &  Bjork,  2008.  p.  106).  The  available  literature 
suggests  general  endorsement  of  the  importance  of  learning  styles.  For  example,  researchers 
argue  that  preferred  learning  styles  vary  between  students,  and  personalized  instruction  can 
capitalize  on  these  differences  to  enhance  learning  for  the  individual  (Andrew  &  Fitzgerald,  2010; 
Cannon-Bowers  &  Bowers,  2008).  Students  find  learning  more  exciting,  easier,  and  consequently 
are  more  motivated  when  they  are  coupled  with  their  unique  learning  style  (see  Andrew  & 
Fitzgerald,  2010;  Meier,  2000).  Unique  attributes  of  the  individual  and  preferred  learning  styles 
of  the  student  can  call  for  learner-centered  instruction.  Meier  (2000)  argues  that  accelerated 
learning  is  the  human  factors  approach  that  brings  the  student  into  the  center  of  learning  which  is 
a  divergence  from  the  traditional  “cookie  cutter”,  lecture -style,  “one  size  fits  all”  teaching 
approach  prevalent  in  western  society.  Interestingly,  Pashler  et  al.  (2008)  argue  that  although 
people  clearly  express  preferences  about  their  own  learning  styles  there  is  little  evidence  that 
different  people  actually  benefit  from  different  teaching  styles  (e.g.,  “visual”  versus  “verbal”). 
This  suggests  that  what  people  prefer  when  learning  is  not  necessarily  what  helps  them  learn 
better. 

Pashler  et  al.  (2008)  identify  the  “meshing  hypothesis”  as  a  common  hypothesis  relevant  to 
learning  styles.  According  to  this  hypothesis,  providing  training  and  education  in  formats  that 
match  the  preferences  of  a  learner  (e.g.,  using  visual  information  for  a  visual  learner)  will 
maximize  learning  effectiveness.  Flowever,  research  by  Pashler  et  al.  (2008)  specifically  focused 
on  the  validity  of  the  meshing  hypothesis.  They  argue  that  understanding  the  impact  of  learning 
styles  requires  carefully  designed  experiments  that  control  for  potential  confounds.  For  example, 
students  would  need  to  be  divided  into  groups  based  on  their  learning  styles,  and  the  groups 
would  be  randomly  selected  to  receive  a  specific  instructional  method.  Credible  evidence  for  the 
true  impact  of  learning  styles  would  be  an  interaction  between  learning  styles  and  instructional 
methods,  such  that  learners  with  varying  styles  would  be  differentially  affected  by  different 
instructional  methods.  Pashler  et  al.  (2008)  argue  that  such  controlled  experiment  research  is  rare, 
and  that  the  research  that  has  been  done  contradicts  the  meshing  hypothesis.  Given  the  lack  of 
strong  empirical  evidence  Pashler  et  al.  (2008)  argue  that  the  widespread  acceptance  of  learning 
styles  as  a  critical  factor  to  consider  in  designing  training  may  be  premature.  They  conclude  that 
“there  is  no  adequate  evidence  base  to  justify  incorporating  learning-styles  assessments  into 
general  educational  practice”  (Pashler  et  al.,  2008.  p.  105).  On  the  other  hand,  this  lack  of 
evidence  does  not  mean  that  learning  styles  do  not  exert  some  impact  -  merely  that  the 
importance  of  learning  styles  has  not  been  scientifically  proven. 
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In  summary,  although  there  is  implicit  agreement  about  the  importance  of  learning  styles,  the 
relevance  of  learning  styles  remains  in  need  of  future  research  and  development. 


2.3.4  Prior  Knowledge  or  Experience 

Prior  knowledge  and  experience  is  also  noted  in  the  literature  to  impact  on  skill  acquisition.  Prior 
knowledge  relevant  to  a  task  or  skill  (e.g.,  previous  training  and  education)  provides  a  foundation 
on  which  to  build  skills  which  can  lead  to  quicker  and  more  effective  learning  and  longer 
retention.  Goodwin  (2006)  reviewed  several  reports  that  indicated  computer  background 
knowledge  has  an  impact  on  training  and  retention  of  digital  skills.  As  expected,  individuals  with 
greater  knowledge  can  more  easily  organize  and  encode  new  information,  as  it  is  more  likely  to 
fit  into  existing  schemas  (Goodwin,  2006).  In  addition,  exposure  to  more  classroom  instruction 
where  systems  are  explained  may  facilitate  learning  of  procedural  skills.  The  theoretical 
background  may  consequently  improve  performance. 

It  is  important  to  point  out  that  prior  knowledge  and  experience  do  not  necessarily  lead  to 
consistently  positive  impacts  on  learning.  For  example,  dissimilar  skills  can  create  interference  or 
negative  transfer  when  attempting  to  learn  new  or  related  skills  (Rowatt  &  Shlechter,  1993). 
Moreover,  prior  negative  experiences  during  learning  can  also  adversely  affect  learning  and 
retention  (Smith-Jentsch,  Jentsch  &  Payne,  1996;  cited  in  Salas  and  Cannon-Bowers,  2001). 

One  notable  type  of  prior  experience  or  knowledge  that  is  important  to  highlight  is  comfort  with 
technology.  Level  of  comfort  with  technology  is  noted  in  the  literature  as  impacting  on  ability  to 
learn  and  perhaps  even  the  motivation  to  learn  (Capuzzi  Simon,  2007;  Zipperer  et  al.,  2003).  In 
this  sense,  students’  negative  previous  experiences  attempting  to  use  unyielding  technology  coidd 
hinder  the  ability  to  learn  in  a  new  situation. 

2.3.5  Other  Student  Attributes 

Several  other  themes  emerged  from  our  review  of  the  literature  as  relevant  student  attributes  that 
might  impact  on  their  learning. 

One  key  characteristic  likely  to  influence  the  ability  to  achieve  accelerated  learning  is  expertise. 
Expertise  has  been  argued  (Hoffman,  1996;  cited  in  Andrews  and  Fitzgerald,  2010)  to  have  3  key 
dimensions.  First,  the  development  of  expertise  is  the  result  of  deliberate  and  persistent  practice 
in  relevant  and  diverse  tasks,  rather  than  being  the  result  of  simple  maturation.  This  means  that 
only  people  who  are  sufficiently  practiced  can  hope  to  have  expertise.  Within  the  available 
literature  (e.g.,  Hoffman  et  al.,  2009),  some  researchers  estimate  that  true  expertise  takes  about  10 
years  to  develop.  Second,  expertise  is  commonly  understood  as  requiring  a  distinct  set  of 
knowledge  structures.  These  structures  are  highly  abstract  and  reflect  deep  understanding  of 
complex  relationships  among  the  various  elements.  These  elements  are  organized  in  ways  that  are 
meaningful  to  the  expert.  Lastly,  because  of  these  rich  knowledge  structures,  experts  are  also  able 
to  anticipate  potential  problems  more  effectively  and  to  solve  problems  that  emerge  more  adeptly 
as  well.  Their  rich  knowledge  allows  them  to  attend  more  efficiently  to  critical  information  and  to 
filter  out  extraneous  information  better  than  novices,  and  they  are  able  to  perform  well  under 
levels  of  high  stress  and  workload  (Hoffman  et  al.,  2009). 
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Researchers  interested  in  accelerated  learning  specifically  focused  on  expertise  in  order  to  create 
programs  that  shorten  the  gap  between  novice  and  expert  performance.  Experts  with  a  deeper 
understanding  of  a  task  seem  to  efficiently  and  effectively  organize  information  into  mental 
models.  The  mental  models  of  experts  are  more  abstract  than  the  models  of  novices,  whose 
mental  models  tend  to  focus  on  surface  features  developed  from  superficial  understanding  of  the 
tasks  (Glaser,  1989;  cited  in  Cannon-Bowers  &  Bowers,  2008;  Schumacher  &  Czerwinski,  1992 
cited  in  Andrews  &  Fitzgerald,  2010;  see  also  Collard,  Gelaes,  Vanbelle,  Bredart,  Defraigne, 
Boniver  et  al.,  2009). 

The  metacognitive  abilities  of  students  also  emerged  from  our  review  of  the  literature. 
Metacognition  is  defined  by  Ford,  Smith,  Weissbein,  Gully,  and  Salas  (1998)  as  the  degree  to 
which  an  individual  is  aware  of  and  in  control  of  his  or  her  cognitions.  The  ability  of  individuals 
to  be  knowledgeable  about  their  own  styles  of  learning  (and  to  be  able  to  adjust  their  efforts 
accordingly)  is  a  key  aspect  of  learning.  Unfortunately,  this  issue  received  passing  mention  rather 
than  discrete  coverage  in  the  limited  number  of  articles  sampled.  For  example,  some  researchers 
noted  that  reflection  on  performance  can  be  used  to  gain  insight  into  errors  and  corrective 
behaviour  (Fu  et  al.,  2009) 

Self-efficacy  is  another  construct  that  has  been  widely  studied  in  relation  to  learning  and  skill 
acquisition.  Self-efficacy  is  defined  as  the  “belief  that  one  can  perform  specific  tasks  and 
behaviours”  (Salas  &  Cannon-Bowers,  2001,  p.  478).  The  power  of  self-efficacy  is  summarized 
in  a  review  by  Salas  and  Cannon-Bowers  (2001,  p.  478)  who  argue  “whether  one  has  it  before  or 
acquires  it  during  training,  leads  to  better  learning  and  performance”.  Self-efficacy  is  another 
characteristic  of  students  that  will  impact  on  their  learning. 

Some  research  noted  in  the  available  literature  also  indicates  that  a  students’  level  of  working 
memory  is  another  important  individual  difference  factor  (Dyer  and  Salter,  2001;  cited  in 
Goodwin,  2006).  This  research  showed  that  presenting  digital  training  by  deliberately  varying 
levels  of  demand  on  working  memory  (in  either  large  chunks  with  9  to  1 8  pieces  of  information, 
or  small  chunks  in  3  to  9  pieces  of  information)  had  significant  impact  on  training  effectiveness. 
Using  lessons  with  small  blocks  of  information  was  more  effective.  This  empirical  study  suggests 
that  attending  to  the  capacity  limitations  of  working  memory  may  be  important  to  promote 
learning. 

2.3.6  Emerging  Research  Themes  and  Questions  -  Student  Attributes 

Many  student  attributes  are  likely  to  impact  on  their  ability  to  learn  information  and  to  retain  it. 
This  section  shows  that  cognitive  ability,  motivation,  learning  styles  and  prior  knowledge  and 
experience  are  key  factors  argued  to  impact  on  the  ability  to  learn  and  to  benefit  from  training. 
Many  individual  attributes  are  generally  less  amenable  to  change.  Although  the  motivation  to 
learn  is  not  necessary  a  stable  trait,  for  example,  cognitive  ability  is  relatively  set  within  boundary 
conditions. 

One  of  the  problems  with  this  area  of  research,  however,  is  that  it  lacks  a  strong  theoretical  base, 
and  there  is  little  evidence  of  truly  integrative  approaches  that  consider  the  role  of  individual 
factors  as  a  whole.  Most  of  the  current  literature  is  limited  to  very  few  factors.  This  area  of 
research  also  lacks  meta-analytic  efforts  that  would  help  provide  critical  information  about  which 
of  the  factors  within  this  area  are  actually  the  most  critical.  At  best,  within  most  of  the  literature 
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reviewed  in  this  report,  individual  differences  are  most  often  considered  in  interactions  with  other 
variables,  such  as  training  variables.  This  makes  it  difficult  to  understand  the  pure  and  distinct 
impact  of  attributes  of  the  individual. 

Perhaps  even  more  critically,  understanding  the  impact  of  individual  attributes  relative  to  other 
sets  of  factors  (such  as  the  impact  of  varying  delivery  media)  is  also  impossible  given  the 
available  literature.  This  means  that  if  research  and  development  efforts  must  target  the  most 
critical  questions  with  the  most  promise  of  “payoffs”  within  a  relatively  short  period  of  time,  it 
would  be  ideal  to  be  able  to  make  a  strong  statement  about  how  important  individual  differences 
really  are.  This  is  a  gap  in  the  current  literature  that  may  need  to  be  addressed. 

Based  on  the  available  literature,  the  two  individual  differences  that  seem  particularly  important 
are  cognitive  ability  and  motivation  of  the  leamer/student.  This  opinion  is  based  on  the 
dominance  of  cognitive  ability  as  a  factor  throughout  the  reviewed  literature,  as  well  as  the 
consistent  prominence  of  motivation.  Motivation  showed  good  presence  in  the  literature  as  well 
as  being  supported  by  substantial  empirical  evidence.  Given  the  relatively  stable  nature  of 
cognitive  ability,  of  course,  the  focus  would  have  to  be  on  how  to  help  learners  with  varying 
levels  of  cognitive  ability  to  make  the  most  out  of  their  capacity.  The  impact  of  motivation  on 
learning  appears  to  offer  a  rich  set  of  research  questions  that  could  be  explored.  Given  that 
motivation  appears  to  have  a  serious  impact  on  how  (and  whether)  individuals  learn,  the  obvious 
question  is  whether  increasing  motivation  can  help  to  accelerate  learning.  However,  the  role  of 
motivation  as  an  interactive  effect  (rather  than  as  a  main  effect)  may  be  more  fruitful,  as  the 
literature  is  strongest  when  motivation  occurs  in  combination  with  other  variables. 

The  potential  role  of  motivation  in  supporting  positive  training  outcomes  is  also  worthy  of  future 
exploration.  One  of  the  possible  stumbling  blocks  for  trainers  is  that  training  can  be  arduous  and 
even  boring  for  students.  Given  the  non-voluntary  nature  of  some  of  the  training  undertaken 
within  a  military  system  such  as  the  CF,  an  important  question  is  whether  motivation  can  be 
meaningfully  enhanced  or  maintained  through  the  construction  of  instruction  techniques  targeting 
student  engagement  and  involvement.  This  is  true  in  terms  of  the  impact  that  the  emergent 
training  technologies  could  have  on  motivation.  Understanding  how  to  use  technology  to  provide 
the  learner  with  a  sense  of  involvement  and  empowerment  would  be  important  contributions  that 
could  have  a  significant  impact  on  learning.  A  key  issue  for  future  research  and  development  to 
explore  is  whether  the  effectiveness  of  training  can  be  meaningfully  affected  by  systematic 
attention  to  tools  and  technologies  that  promote  learner  motivation. 

The  research  reviewed  in  this  section  shows  that  the  unique  attributes  of  individuals  have  a  clear 
impact  on  the  ability  to  learn.  Clear  and  unequivocal  information  about  the  boundary  conditions 
of  individual  attributes  in  a  range  of  training  environments  and  given  a  range  of  diverse  tasks 
would  be  helpful.  Specific  findings  and  possible  research  questions  are  shown  in  Table  4. 


Table  4  Findings  and  Research  Questions-  Student  Attributes 


Student  attributes  impact  on  learning 

What  student  attributes  are  most  critical  to  learning? 

What  is  the  relative  importance  of  student  attributes  relative  to 
other  sets  of  factors  (e.g.,  delivery  media)? 

Cognitive  ability  promotes  learning  (Salas  & 

How  can  learning  be  optimized  for  individuals  with  low  cognitive 
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Cannon-Bowers,  2001) 

abilities? 

High  motivation  to  learn  increases  learning  (Blume, 
Ford,  Baldwin,  &  Huang,  2010;  Meier,  2000). 
Hands-on  experience  can  increase  motivation. 
Understanding  relevance  of  training  can  increase 
motivation  to  learn  application  of  newly  learned 
skills.  (TLC,  2010;  Zipperer,  Klein,  Fitzgerald, 

Kinnison,  &  Graham,  2003). 

How  can  training  be  designed  to  maximize  student  motivation? 
What  approaches  and  technologies  (e.g.  virtual  worlds)  best 
promote  immersion,  engagement,  social  presence,  how  best  to 
promote  “flow” 

Embodiment,  presentation  and  engagement  (physical 
representation  of  the  user,  e.g.,  fidelity  of  the  avatar) 

Learning  styles  may  influence  learning 

Should  learning  styles  be  further  investigated? 

Prior  knowledge  or  experience  can  improve  learning 
(e.g.,  computer  background  knowledge  has  an 
impact  on  training  and  retention  of  digital  skills 
(Goodwin,  2006). 

How  can  prior  knowledge  and  experience  be  brought  to  bear  on 
training  effectiveness? 

Level  of  comfort  with  technology  can  slow  or 
increase  learning  and  motivation  to  learn  when 
using  technological  tools  (Capuzzi  Simon,  2007; 
Zipperer  et  al.,  2003)Expertise  improves  learning 
(Andrews  &  Fitzgerald,  2010) 

How  does  experience  and  comfort  with  technology  impact  on 
learners  in  technology-based  training?  What  compensatory 
approaches  might  support  personnel  with  low  levels  of 
experience? 

How  can  training  build  expertise  more  quickly? 

How  can  expertise  best  be  leveraged? 

Can  emerging  technologies  support  the  emergence  of 
expertise? 

2.4  Instructor  Attributes 

Instructors  have  a  key  role  to  play  in  training.  Instructors  are  obviously  very  diverse,  and  factors 
such  as  their  subject  matter  expertise,  their  experience  as  a  teacher  and  their  style  of  teaching 
emerged  from  the  literature  review  as  important  instructor  attributes.  Within  the  work 
environment,  a  meta-analysis  by  Blume  et  al.  (20 1 0)  found  that  support  (peer  and  supervisor)  was 
the  most  influential  factor  on  transfer  of  training.  Instructors,  then,  have  a  potentially  critical  role 
in  promoting  learning  and  training  effectiveness. 

Many  training  programs  use  experts  to  provide  instruction  to  students;  military  programs  have 
also  taken  this  approach  (Zipperer  et  al.,  2003).  Learning  from  high  performing  training 
specialists  who  know  the  material  in  question  and  can  effectively  provide  instruction  may  be 
“more  important  than  ever”  if  the  goal  is  to  promote  a  high  degree  of  learning  and  retention 
(Zipperer  et  al.,  2003,  p.  22).  However,  one  potential  downside  to  using  experts  to  present 
training  is  that  their  subject  matter  expertise  does  not  necessarily  make  them  good  instructors. 
Experience  as  an  instructor  is  another  important  variable  to  consider.  The  motivation  of  an 
instructor  to  teach  is  another  possible  influence  on  the  effectiveness  of  learning. 

The  role  of  mentors  in  promoting  learning  is  also  relevant  to  this  discuss.  A  mentor  may  or  may 
not  be  an  instructor,  but  they  share  some  of  the  role  of  instructors,  namely  to  help  promote  the 
learning  of  students  or  peers.  Hoffman  et  al.  (2009)  identify  several  attributes  of  a  good  mentor, 
as  follows: 

•  Able  to  create  appropriate  content  for  the  learner 

•  Able  to  form  a  mental  model  of  the  learner’s  knowledge  and  skill 

•  Able  to  anticipate  probable  errors,  and  to  tailor  exercises  accordingly 
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Although  these  attributes  are  most  easily  applied  when  the  mentor  is  a  human  instructor,  these 
attributes  are  also  requirements  that  must  be  met  by  intelligent  systems  that  are  specifically 
designed  to  challenge  the  learner. 

In  sum,  the  role  of  the  instructor  requires  a  combination  of  delegation,  support,  and  coaching  that 
is  matched  to  student  commitment  and  capability  related  to  the  task  at  hand. 

2.4.1  Emerging  Research  Themes  and  Questions  -  Instructor  Attributes 

The  instructor  has  also  been  identified  as  a  key  element  in  training  effectiveness.  The  issue  of 
how  instructors  within  the  CF  training  system  can  be  most  effective  would  be  an  important  issue 
for  future  research  and  development.  Within  the  CF  training  system,  instructors  come  from  all 
levels  and  specialties,  and  the  level  of  their  qualifications  as  subject  matter  experts  and  instructors 
varies  widely.  An  important  issue  for  future  research  and  development  efforts  would  be 
exploration  of  the  impact  of  varying  levels  of  instructor  expertise  and  experience  as  instructors.  A 
number  of  sample  research  questions  that  could  focus  this  research  are  provided  in  Table  5. 


Table  5  Findings  and  Research  Questions-  Instructor  Attributes 


Instructor  Attributes 

Instructor  attributes  (e.g.,  subject  matter 
expertise,  teaching  experience  and  teaching 
style)  impact  on  learning 

What  are  the  most  critical  aspects  of  instructor 
effectiveness? 

What  is  the  ideal  mix  of  subject  matter  expertise  and 
teaching  experience? 

What  makes  a  good  mentor? 

2.5  Delivery  media 

Advancements  in  technology  over  the  past  20  years  have  yielded  a  range  of  literature  and 
research  aimed  at  improving  learning  and  retention  through  the  use  and  implementation  of 
technology  in  training.  Technology  may  be  particularly  useful  in  circumstances  where  time 
resources  are  limited  and  brief  episodes  of  skill  performance  can  provide  some  form  of  practice 
that  will  help  to  retain  competency  (Andrews  &  Fitzgerald,  2010).  This  section  describes  various 
delivery  media  identified  in  the  literature  review. 

2.5.1  Classroom  Aids  and  Environment 

2. 5.1.1  Conventional  Classroom  Instruction 

Conventional  classroom  instruction  has  been  the  dominant  way  to  promote  learning  and  retention. 
Stereotypic  classrooms  consist  of  an  instructor,  students  and  typical  means  of  classroom 
instruction  include  blackboards  and  other  media  devices.  There  are  obvious  many  different 
methods  and  approaches  related  to  the  conduct  of  classroom  construction.  For  example 
(Goodwin,  2006,  p.  5)  cites  the  work  of  Clark  and  Wittrock  (2000),  who  conceptualized  training 
in  terms  of  digital  or  non-digital  and  the  continuum  of  instruction  as  ranging  “from  external 


14 


DRDC  Toronto  CR  201 1-105 


(instruction  is  driven  by  environmental  factors)  to  internal  (driven  by  factors  internal  to  the 
learner)”.  Four  types  of  training  along  this  continuum  are: 

•  Receptive  (teaching  by  telling) 

•  Behavioural  (teaching  by  demonstration,  practice  and  feedback) 

•  Guided  discovery  (teaching  by  problem  solving) 

•  Exploratory  (teaching  by  exploration) 

Interestingly,  Clark  and  Wittrock  (2000;  cited  in  Goodwin,  2006)  argue  that  there  may  be  no 
uniformly  best  approach,  but  that  this  depends  on  the  best  match  for  the  individual.  For  example, 
they  argue  that  receptive  approaches  may  be  best  for  novices,  but  that  experts  with  the  motivation 
and  metacognitive  awareness  may  do  better  with  exploratory  approaches. 

In  general,  there  is  some  evidence  that  training  that  consists  only  of  receptive  style  of  instruction 
may  not  be  optimal.  Lecture -based  instruction  is  argued  to  be  less  favourable  by  many  students 
(Schaab  &  Dressel,  2001)  and  ineffective  in  terms  of  accelerating  learning  (Meier,  2000).  Other 
exploratory  techniques  that  provide  experiential  learning  have  been  argued  to  support  accelerated 
learning  (Meier,  2000).  In  the  end,  the  most  effective  type  of  instruction  may  depend  on  the 
training  content,  student  attributes,  and  available  resources. 

2. 5.1. 2  Learning  Environments 

Creating  a  positive  learning  environment  is  essential  in  accelerated  learning  (Meier,  2000). 
Accelerated  learning  is  more  likely  to  occur  within  an  enjoyable  training  environment.  A  positive 
learning  environment  is  argued  to  increase  learning  and  makes  accelerated  programs  more 
efficient  (Meier,  2000). 

In  a  short  report,  Training  &  Leadership  Coaching  (TLC)  (2010)  emphasized  that  a  warm, 
welcoming,  comfortable  environment  was  most  effective  in  fostering  learning.  Positive  learning 
environments  can  be  achieved  through  attention  to  physical  and  psychological  elements. 

The  physical  aspects  of  a  room,  the  layout,  lighting,  views  from  the  window,  access  to  fresh  air, 
and  space  to  move  around  are  important  aspects  of  a  comfortable  learning  environment  and 
should  be  considered  in  part  of  the  planning  process  of  a  training  program. 

Engaging  all  five  senses  and  involving  students  in  different  activities  that  make  the  learning 
experience  enjoyable  while  reinforcing  content  is  the  foundation  of  accelerated  learning  (Meier 
2000;  TLC,  2010).  Meier  (2000)  detailed  the  importance  of  engaging  the  learner  through  auditory 
(e.g.,  playing  music,  engaging  in  discussion,  reading  aloud),  visual  (e.g.,  skits,  video,  images), 
kinaesthetic  (e.g.,  active  games,  field  trips,  performing  tasks),  and  olfactory/gustatory  techniques 
(e.g.,  aroma  therapy,  extinguishing  negative  aromas,  pairing  an  aroma  or  flavour  with  learning). 
The  justification  for  stimulating  multiple  senses  in  such  a  way  is  that  students  learn  more  when 
fully  immersed.  We  learn  more  French  when  in  France,  fully  surrounded  by  French  language  and 
culture  including  music,  signs,  smells,  and  food  (Meier,  2000).  Another  example  described  by 
Erickson  (1991)  showed  that  playing  music  during  training  or  during  breaks  increased  students’ 
moods  resulting  in  better  learning  and  retention.  Caution  is  also  advised  in  over-use  or 
inappropriate  use  of  music.  For  example,  different  tastes  in  music  may  make  a  learning 
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environment  unfavourable  to  some,  and  music  should  not  be  used  unless  all  students  agree  on  the 
music  to  be  played.  From  a  training  perspective,  the  key  is  that  instructors  should  actively  think 
about  the  impact  of  adding  sense-stimulators  to  the  learning  environment.  They  should  be  applied 
with  caution  and  if  something  is  attempted  that  does  not  provide  the  desired  effects,  it  should  be 
stopped  or  altered. 

In  contrast  to  a  focus  on  the  physical  environment,  psychological  components  of  the  environment 
also  impact  learning.  Learning  culture  (or  similarly,  transfer  climate)  is  the  sum  of  attitudes, 
values,  and  norms  that  stem  from  supervisory  and  peer  support,  task  cues,  training  accountability, 
opportunities  to  practice,  opportunities  to  use  new  knowledge  and  skills,  and  intrinsic  and 
extrinsic  rewards  for  application  of  knowledge  (see  Aguinis  &  Kraiger,  2009).  A  recent  meta¬ 
analysis  by  Blume  et  al.  (2010)  shows  the  importance  of  the  work  climate  as  an  impact  on 
transfer  of  training.  This  review  showed  transfer  climate  as  being  moderately  correlated  with 
transfer.  However,  Aguinis  &  Kraiger  (2009)  have  argued  that  research  relevant  to  transfer,  as  a 
whole,  is  mixed  and  inconsistent.  They  reviewed  studies  showing  both  positive  (e.g.,  Richman 
and  Hirsch,  2001;  cited  in  Aguinis  &  Kraiger,  2009)  and  null  effects  (e.g.,  van  der  Klink  et  al., 
2001;  cited  in  Aguinis  &  Kraiger,  2009)  related  to  positive  transfer  climate. 

Although  this  area  of  research  has  some  mixed  results,  there  is  agreement  in  the  literature  about 
the  importance  of  positive  learning  environments. 

2. 5. 1.3  Classroom  Aids 

Several  classroom  aids  that  can  be  used  to  promote  learning  also  emerged  from  the  literature. 
Classroom  aids  are  identified  as  an  activity  based  tools  or  technique.  Those  described  in  the 
literature  include: 

•  Concept  mapping  (e.g.,  Panel  and  flow  drawings) 

•  Opportunity  training 

•  In-class  tools  and  techniques  (e.g.,  audio  tape  piping  and  instrumentation  ,  or  P  &  ID, 
Crossword  puzzles) 

Concept  mapping 

Concept  maps  or  “mind  maps”  can  be  used  to  form  explicit  knowledge  frameworks  of  new 
information  (Erickson,  1991).  Expert  mental  models  are  more  abstract  than  those  of  novices. 
Making  mental  models  explicit  through  tools  such  as  mind  maps  are  argued  to  promote  learning 
and  advance  expertise  (Erickson,  1991).  Creating  drawings  of  mental  models  can  accelerate 
learning  through  promoting  both  differentiated  and  integrated  organization  of  the  material  being 
learned.  Besides  mind  maps,  concept  maps  can  be  displayed  as  panel  and  flow  drawings, 
placards,  posters,  or  flipcharts  that  can  be  developed  by  students  during  a  learning  program  and 
later  used  as  a  resource  to  reflect  on  the  material  learned  (Erickson,  1991).  Mental  models  can  be 
developed  and  reminders  of  learned  material  may  serve  to  ‘refresh’  the  learned  material. 

Concept  mapping  techniques  or  tools  can  be  implemented  differently  depending  on  the  context, 
course  material,  student  and  instructor  preferences,  and  available  resources.  Techniques  should  be 
selected  and  implemented  based  on  the  intended  goal.  For  example,  mind  maps  have  been  found 
to  be  effective  in  forming  explicit  knowledge  networks  of  information.  Mind  maps  may  be 
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preferable  in  circumstances  where  the  goal  is  to  help  students  learn  and  practice  how  to  develop 
mental  models. 


Opportunity  training 

Opportunity  training  is  one  way  to  maximize  time  efficiency,  learning,  and  retention  that  are 
fundamental  in  military  training  (see  Erickson,  1991  and  Zipperer  et  al.,  2003).  A  recent  review 
investigated  training  techniques  useful  within  military  training,  specifically  instruction  of 
dismount  combatants  and  small  units  (Zipperer  et  al.,  2003).  A  relevant  technique  identified  was 
“opportunity  training”  or  “hip-pocket”  training,  where  instructors  conducted  short  15-20  minute 
lessons  between  the  “cracks”  of  a  training  program  (e.g.,  compass  training  during  a  water  break 
while  on  a  hike).  Opportunity  training  can  be  used  to  decrease  the  hours  within  a  training 
program;  unfortunately,  resources  required  for  such  training  are  not  always  available.  Printed 
copies  of  manuals  or  soldier  handbooks  that  are  often  used  are  no  longer  printed,  or  are  rare  and 
thus  have  limited  the  amount  that  this  method  is  currently  implemented.  There  was  no  evidence 
as  to  how  much  this  technique  has  been  used  or  whether  it  has  been  shown  to  be  effective. 


In-class  tools  and  techniques 

Some  additional  aids  were  noted  during  the  literature  review  (Erickson,  1991),  but  received  only 
limited  attention,  as  follows: 

1 .  Audio  tape  piping  and  instrumentation  (P&1D)  has  been  used  as  a  method  of  making 
training  unique  and  different  helping  to  capture  students’  interest  and  provide  motivation. 
Caution  is  advised  not  to  overuse  this  technique  (Erickson,  1991). 

2.  Crossword  puzzles  are  an  activity  that  can  provide  a  fun  method  of  learning,  maintain 
student’s  interest  and  accelerate  learning  (see  Erickson,  1991). 

3.  Games  provide  novel  and/or  motivating  methods  of  learning  (Erickson,  1991;  Cannon- 
Bowers  &  Bowers,  2008).  Technology  has  been  used  to  provide  gaming-based  training 
programs  and  its  application  is  expected  to  continue  to  grow  in  the  future. 

2.5.2  Distance  Learning 

Distance  learning  emerged  from  the  literature  review  as  an  important  delivery  media.  The 
Merriam- Webster  dictionary  defines  distance  learning  as  “Education  that  takes  place  via 
electronic  linking  instructors  and  students  who  are  not  together  in  a  classroom”.  Students  are 
often  geographically  separated  from  each  other  as  well  as  from  the  instructor.  It  is  possible 
however,  to  have  several  students  co-located  with  a  geographically  distributed  instructor,  or  to 
have  several  students  and  an  instructor  co-located  while  other  students  are  geographically 
distributed.  Industry  and  education  are  the  largest  sectors  in  society  that  utilize  distance  learning 
(Braddock,  Berryman,  Dickinson,  Gerry,  Elartman,  et  al..  1997).  Braddock  et  al.  (1997)  point  out 
that  within  the  educational  system,  distance  learning  courses  are  developed  based  on  principles  of 
making  learning  efficient  and  effective  for  students  to  acquire  knowledge. 

There  are  multiple  ways  of  implementing  distance  learning.  Braddock  et  al.,  (1997)  describe  three 
distance  learning  teaching  modes.  Distance  learning  can  be  1)  remote  requiring  synchronous 
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attendance  by  students  and  instructors;  2)  canned  in  which  case  material  is  recorded  and  accessed 
by  students  individually  at  a  convenient  time  for  them  (asynchronous);  and  3)  collaborative 
where  students  can  interact  through  discussion  boards,  emails  (asynchronous)  and  or  chat 
(synchronous).  These  techniques  are  not  necessarily  mutually  exclusive,  and  a  combination  of 
them  may  be  optimal  depending  on  type  of  skill/knowledge  to  be  trained,  resources  allocated  to 
the  instructor,  resources  of  the  student,  and  additional  responsibilities  either  may  carry  (e.g., 
career  or  family  obligations). 

There  is  some  evidence  in  the  available  literature  about  the  effectiveness  of  distance  learning  for 
promoting  learning.  A  report  in  the  New  York  Times  details  evidence  from  U.S.  universities  that 
shows  students  who  enrol  in  a  course  that  has  a  component  of  distance  learning  earn  higher 
grades  compared  with  those  who  enrol  in  residential  courses  (Capuzzi  Simon,  2007).  Particularly 
for  adult  education,  distance  learning  provides  flexibility  and  mobility  needed  to  juggle  training 
and  education  with  careers  and  family  life  (Capuzzi  Simon,  2007).  Learner  responsibility  can  be 
increased  by  providing  students  with  control  over  tasks  performed  or  by  providing  the  student 
with  control  over  learning  such  as  controlling  the  pace  of  learning  as  is  often  seen  in  distance 
education  or  on-line  courses  (Cannon-Bowers  &  Bowers,  2008). 

As  technology  advances,  the  tools  available  for  promoting  distance  learning  have  also  increased. 
For  instance,  instead  of  stationary  desk  computers,  mobile  (portable)  technologies  such  as  iPads 
are  increasingly  being  used  to  promote  mobile  learning  (m-leaming).  Mobile  media  may  be 
effectively  utilized  particularly  in  a  military  environment  when  it  may  not  be  feasible  or 
convenient  for  trainees  to  access  computer-based  instruction  via  computer  laboratories,  offices,  or 
home -based  computers.  Mobile  technology  is  also  increasing  the  use  of  multimedia  in  both 
formal  (classroom  based)  and  informal  (peer  discussions)  learning.  Today’s  technological  culture 
has  increased  the  opportunities  for  informal  learning.  Having  access  to  portable  devices  to 
communicate,  play  online  games,  search  the  internet,  and  read  internet  based  material  may 
provide  many  more  opportunities  to  learn  informally  than  available  through  formal  means  such  as 
classrooms.  This  may  be  one  area  that  requires  further  exploration,  specifically  focusing  on 
technology  and  computer-based  learning. 

Nonetheless,  distance  learning  is  also  noted  as  having  some  potential  limitations.  For  example, 
distance  learning  restricts  the  level  of  interactivity  between  the  student  and  instructor.  This 
suggests  that  distance  learning  courses  may  be  most  effective  for  students  who  are  motivated  and 
disciplined,  rather  than  those  who  require  more  hands-on  guidance  from  the  instructor.  As  noted 
in  earlier  sections,  learning  styles  and  leadership  styles  can  affect  learning. 

2.5.3  Simulator-Based  Instruction 

Simulations  are  predominantly  used  to  training  new  skills  and,  but  some  researchers  have  argued 
that  they  have  been  rarely  used  in  refresher  training  (Ginzburg  &  Dar-El,  2000).  Similarly, 
simulation-based  research  has  investigated  immediate  performance  effects  and  little  is  known 
about  the  effects  on  retention. 

A  review  of  the  literature  showed  that  simulator-based  instruction  has  focused  on  synthetic 
learning  environments  (SLE).  SLEs  are  technology-based  applications  that  work  to  reflect  real 
world  experiences  (i.e.,  simulations,  games,  and  virtual  worlds)  which  can  provide  unique 
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learning  opportunities  (e.g.,  Cannon-Bowers  &  Bowers,  2008).  Several  advantages  of  using  SLEs 
for  training  include: 

•  providing  a  collaborative  tool  for  geographically  distributed  team  members, 

•  providing  real  world  experiences  when  actual  experiences  are  not  feasible  (e.g., 
equipment  is  not  available),  not  practical  (e.g.,  too  dangerous,  theatre  of  operation  is  not 
accessible),  or  infrequent  (e.g.,  emergency  procedures), 

•  providing  feedback  during  or  immediately  after  task  completion, 

•  providing  cost  savings  relative  to  training  on  operational  equipment, 

•  allowing  for  “tremendous  flexibility  in  manipulations  of  fidelity  -  differing  kinds  and 
amounts  of  contextual  richness”  (Hoffman  et  al.,  2009,  p.  21). 

There  is  good  evidence  of  the  effectiveness  of  SLEs  in  education.  Research  has  shown  that  using 
SLEs  to  train  college  or  high  school  subjects  (e.g.,  visual  attention  skills,  programming  skills, 
geology,  and  cell  biology)  increases  student  outcomes  (see  Green  &  Bavelier,  2003,  Emurian, 
2005,  and  McClean  et  ah,  2001;  cited  in  Cannon-Bowers  &  Bowers,  2008).  Research  using  SLEs 
has  demonstrated  that  web-presented  information  significantly  increased  test  scores  of  students 
compared  with  those  in  more  typical  lecture  (control)  groups  (Cannon-Bowers  &  Bowers,  2008). 

However,  one  serious  gap  in  the  literature  has  been  identified.  As  Cannon-Bowers  and  Bowers 
(2008,  p.  318)  have  noted,  although  there  has  been  increasing  reliance  in  technology  over  the  past 
30  years  “...some  of  these  development  efforts  have  been  successful,  others  have  not,  and  still 
others  (the  majority  perhaps)  are  of  unknown  value  because  they  have  never  undergone  rigorous 
empirical  testing”.  Moreover,  they  also  argue  that  “SLEs  are  almost  always  built  with  an 
emphasis  on  technology,  and  pedagogy  is  often  worked  in  as  an  afterthought.”  (2008,  p.  319). 
Nonetheless,  they  argue  that  because  of  what  is  known  about  how  people  learn  (e.g.,  experts  tend 
to  chunk  information,  and  build  integrated  mental  models  to  be  able  to  do  so),  it  is  clear  that  SLEs 
should  help  to  promote  learning. 

In  military  contexts,  the  use  of  technology  for  training  is  becoming  increasingly  common.  The 
effectiveness  of  three  technological  training  techniques  used  in  an  established  command  and 
control  curriculum  at  Army’s  Armour  School  at  Fort  Knox,  Kentucky  were  compared  by  Fu  and 
colleagues  (2009)  to  highlight  the  benefits  of  each  approach.  These  techniques,  live 
demonstration,  terrain  board  demonstration,  and  virtual  demonstration,  were  used  for  soldier 
training. 

A  live  demonstration  involving  a  breaching  exercise  (employing  tactics,  techniques  and 
procedures  to  overcome  obstacles  that  stop,  delay,  divert  or  restrict  movement  such  as  in  a 
convoy),  was  conducted  with  all  instructors  and  students  present.  Exercises  were  performed  in 
real  time  and  tasks  were  conducted  by  each  student.  A  vast  amount  of  resources  were  required 
including  vehicles,  fuel,  and  collocation  of  all  students  and  instructors.  Feedback  was  provided 
post  hoc  in  After  Action  Reviews  (AARs). 

Terrain  board  demonstrations  used  PowerPoint  slides  to  guide  instruction  and  allow  instructor 
facilitated  discussion  and  collaboration  (peers  working  together  and  sharing  experiences)  during 
critical  reflection  and  discussion.  In  addition,  soldiers  were  provided  with  the  opportunity  to 
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practice  command  and  control  tasks  using  a  3-D  model  terrain  board.  The  ability  to  collectively 
share  experiences  and  practice  repeatedly  while  receiving  immediate  feedback  from  peers  and 
instructors  were  highlighted  as  advantages. 

Virtual  demonstration  provided  a  walk-through  visualization  of  the  tasks  controlled  by  the 
instructor.  Viewpoints  (camera  angle)  and  time  (duration  of  the  lesson)  can  be  varied  and 
controlled  by  the  instructor.  As  a  virtual  tool,  it  can  provide  a  lot  of  flexibility.  Terrain  can  be 
easily  and  quickly  switched  and  cross-training  is  possible  as  both  the  instructor  and  student  can 
operate  or  perform  tasks  within  the  environment.  Collocation  is  not  necessary  if  the  instructor  and 
students  are  equipped  with  appropriate  resources  while  geographically  distributed.  As  a  cost 
benefit,  the  lesson  can  be  recorded  and  later  played  back  for  review,  practice,  and/or  retraining. 

Results  showed  that  both  the  terrain  board  demonstration  and  the  virtual  world  demonstration 
could  be  conducted  in  faster  than  real-time.  Compressed  training  that  leads  to  high  levels  of 
proficiency  demonstrate  accelerated  learning.  The  positive  performance  outcomes  shown  in  this 
study  are  evidence  of  the  effective  application  of  technology  to  accelerated  learning  and  improve 
learning.  A  comparison  summary  of  all  three  training  approaches  is  shown  in  Figure  1. 


Live  Demonstration 

Terrain  Board 
Demonstration 

Virtual  Demonstration 

Viewing  Perspective 

Real,  situated,  tactical 

Model,  2-D  bird’s  eye  view 

Both  2-D  bird's  eye  and  3-D 
tactical  views,  either 
instructor-  or  trainee-driven 

Resources 

Live  use  of  real  assets 

Small-scale  model  terrain 
construction 

Computer-based  terrain 
construction 

Scene  Arrangement 

Longest,  involving  physical 
placement 

Fast,  requiring  placement  of 
micro  armor  assets 

Fast,  virtual  world 
placement  of  assets 

Time  Efficiency 

Real-time 

Faster  than  real-time 

Faster  than  real-time 

Shared  Experience 

Yes  (team  -  no) 

Yes 

Potentially  yes 

Repeatable 

Yes,  requires  moving  assets, 
role  players 

Yes,  requires  moving  assets 

Yes  -  with  simulation 
control 

Stimuli 

Real  life 

Abstraction 

Life  like  +  abstraction 

Feedback 

Delayed 

Immediate 

Immediate  with  playback 

Team  Training  Capable 

Yes  -  limited  cross-training 

Yes  -  limited  cross-training 

Yes 

Distribution 

No  -  All  participants  must 
be  present 

No  -  All  participants  must 
be  present 

Potentially  yes 

Figure  1  Comparison  of  Armour  School  demonstration  approaches  (Fu  et  al.,  2009) 

The  research  by  Fu,  et  al.  (2009),  undertaken  for  the  U.S.  Army  shows  the  dilemma  presented  by 
the  current  state  of  the  literature,  “While  most  will  agree  that  live  demonstrations  are  the  most 
effective  way  to  convey  information  to  warfighters,  this  conjecture  has  never  been  proven”  (p.  1). 
They  argue  further  that  demonstrations  can  be  undertaken  in  typical  ways  or  in  virtual  world 
simulations  but,  “what’s  needed  is  a  framework  that  can  taxonomize  training  demonstrations,  and 
prescribe  ways  to  measure  the  usefulness  of  any  given  demonstration”  (Fu  et  al.,  2009,  p.l). 

Cannon-Bowers  and  Bowers  (2008)  note  the  potential  value  of  using  scenarios  (such  as  in 
demonstration  type  training),  and  also  point  out  challenges  noting  “One  of  the  problems  with 
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scenario  or  case  design  is  that  it  is  typically  very  time  consuming  and  requires  expertise  from 
SMEs  and  instructional  designers.  A  possible  solution  to  this  challenge  is  to  automate  the 
development  of  scenarios  by  providing  authoring  tools  for  users;  development  of  such  tools 
should  be  a  priority  in  future  work”  (Cannon-Bowers  &  Bowers,  2008,  p.  323). 

As  demonstrated  by  Fu  et  al.,  (2009),  simulations  can  be  used  to  develop  realistic  training  where 
learning  and  practice  activities  closely  reflect  real-world  performance  requirements  (see  also 
Zipperer  et  al.,  2003).  Simulations  can  be  used  in  a  manner  to  challenge  individuals  where  real- 
world  problems  are  introduced  frequently  (relative  to  real-time)  in  training.  Hoffman  et  al.  (2009) 
note  that  simulations  should  be  constructed  to  intentionally  mislead  a  student  seeking  solutions  to 
specific  problems. 

“People  must  be  trained  to  be  resilient,  so  that  they  can  cope  with  complexity  when 
unexpected  events  stretch  resources  and  capabilities.  And  people  must  be  trained  faster. 
Intelligent  systems  technology,  and  intelligent  use  of  technology,  will  certainly  play  a 
critical  and  perhaps  necessary  role  in  this”.  (Hoffman  et  al.,  2009,  p.  21). 

The  situations  and  methods  in  which  simulation-based  instruction  can  be  used  are  vast.  Salas  and 
Cannon-Bowers  (2001)  argue  that,  in  general,  there  is  strong  empirical  evidence  that  simulators 
can  be  an  effective  tool;  however  they  note  that  SLEs  have  not  been  found  to  be  universally 
effective.  “Precisely  why  simulation  and  simulators  work  is  not  well  known.  A  few  studies  have 
provided  preliminary  data...,  but  there  is  a  somewhat  misleading  conclusion  that  simulation  (in 
and  of  itself)  leads  to  learning”  (Salas  &  Cannon-Bowers,  2001,  p.  484). 

Other  critics  have  warned  against  potential  negative  effects  of  training  using  simulations  as  it 
requires  individuals  to  spend  time  learning  how  to  use  simulation  equipment,  and  often  includes 
learning  to  work  as  a  collective  team  (Zipperer  et  al.,  2003).  Though  the  latter  is  vital  to  effective 
team  work,  it  takes  time  away  from  training  individual  skills  that  must  be  mastered  in  order  to 
support  the  collective  (Zipperer  et  al.,  2003). 

In  conclusion,  simulation-based  instruction  has  been  demonstrated  as  very  useful  in  training. 
Specifically,  it  has  been  used  to  compress  training  and  engage  students.  As  noted,  however, 
exactly  why  simulation  is  effective  at  promoting  learning  outcomes  does  not  appear  to  be 
sufficiently  well  understood. 

2.5.4  Computer-Based  Instruction 

This  section  details  the  nature  of  computer-based  instruction,  tools  that  have  been  implemented  in 
such  training,  possible  implications  to  the  instructor,  and  digital  skills. 

Andrews  and  Fitzgerald  (2010)  point  out  that  in  order  to  develop  accelerated  learning  programs, 
non-time  resources,  such  as  computer-based  instruction,  will  be  required.  Computer-based 
instruction  can  involve  a  range  of  media  from  simple,  non-interactive  videos  to  highly  interactive, 
multimedia,  multi-player,  collaborative  gaming  environments.  They  can  also  be  highly  structured 
with  only  one  path  to  follow  through  learning  material,  or  loosely  structured  allowing  trainees  to 
explore  at  will.  With  the  increasing  evolution  of  technology,  the  role  of  computer-based 
instruction  is  expanding  the  possibilities  of  training,  and  associated  research  areas. 
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Simplistic  computer-based  instruction  may  include  videos,  multiple  choice  activities,  or  single¬ 
task  practice  activities  (e.g.,  typing  tutor).  More  advanced  tutoring  programs  can  have  varying 
levels  of  intelligence.  Cognitive  Tutoring  Systems  provide  a  technological  framework  which 
allows  retrieval,  practice,  and  provides  adaptive  feedback  (see  Andrews  &  Fitzgerald,  2010).  The 
Graphical  Instruction  in  LISP  (GIL)  is  one  such  system  that  presents  lessons  and  exercises  as  well 
as  monitored  and  analysed  student  progress.  The  system  provides  advantages  such  as 
individualized  feedback  and  is  able  to  adjust  lessons  based  on  demonstrated  proficiency  (Bryant 
&  Angel,  2000). 

Another  advantage  to  computer-based  instruction  is  the  ability  to  record  trainees’  progress 
throughout  the  task,  such  as  with  the  intelligent  tutoring  systems.  Baseline  measures  can  be  easily 
obtained  and  follow-on  practice  can  focus  on  weak  areas.  Accurate  targeting  of  weak  areas  for  the 
purpose  of  remediation  may  be  one  method  that  effectively  leads  to  accelerated  learning. 

Highly  interactive  computer-based  instruction,  such  as  gaming,  can  provide  learning 
opportunities  that  offer  more  time  in  practicing  a  task,  and  increase  motivation  and  engagement 
(Andrews  &  Fitzgerald,  2010).  Examples  of  such  games  include  Immune  attack,  America’s 
Army:  Special  Forces,  Spore,  and  River  City  (Andrews  &  Fitzgerald,  2010).  Some  research  has 
been  done  that  indicates  that  the  use  of  gaming  can  lead  to  accelerated  learning.  For  example, 
Cannon-Bowers  and  Bowers  (2008)  demonstrated  that  gaming-based  simulations  significantly 
increased  test  scores  compared  with  control  (lecture)  groups.  Thus,  computer-based  instruction 
that  provides  convenience  and  engagement  of  the  leamer(s)  can  promote  positive  learning 
outcomes. 

However,  there  is  also  some  evidence  that  this  form  of  instruction  can  also  have  a  significant 
impact  on  the  instructor.  Development  of  training  programs  that  use  computer-based  instruction 
can  substantially  reduce  the  amount  of  time  needed  for  instruction  (Kulik  &  Kulik,  1991).  Having 
basic  assessments  of  performance  automated  within  a  tutoring  system  would  also  reduce  some 
obligation  on  the  part  of  the  instructor  to  provide  such  assessments.  However,  depending  on  the 
nature  of  the  computer-based  instruction,  there  may  be  more  up  front  work  required  to  prepare 
and  plan  training;  a  responsibility  that  would  most  likely  fall  on  the  instructor.  It  is  worth  noting 
however,  that  once  planned,  the  computer-based  program  may  be  used  and  reused  without 
additional  planning,  thus  making  the  upfront  work  cost  effective  in  the  long-run. 

Digital  skills  are  relevant  to  computer-based  instruction.  They  are  defined  as  related  to  the  use  of 
software  on  a  computer,  requiring  some  form  of  data  entry  through  a  graphical  user  interface 
(GUI)  (Goodwin,  2006),  and  they  range  from  being  relatively  procedural  in  nature  (e.g.,  clicking 
a  button)  to  managing  highly  complex  systems  (e.g.,  using  multiple  digital  systems  for  battlefield 
command  and  control).  Training  digital  skills  will  be  an  important  part  of  training  as  instruction 
and  learning  become  increasingly  computer-based.  Research  by  Goodwin  (2006)  identifies 
several  patterns  evident  in  the  literature  relevant  to  the  effectiveness  of  digital  training.  First,  he 
argues  that  unguided  exploration  is  the  least  effective  way  to  train  digital  skills.  Second,  digital 
training  that  uses  at  least  some  form  of  behavioural  modelling  (with  either  live  or  videotaped 
demonstrator)  is  more  effective  than  a  fully  computerized  tutorial.  However,  this  pattern  of  results 
is  true  for  acquisition,  but  not  retention.  Lastly,  Goodwin  (2006)  argues  that  guided  exploration 
(e.g.,  minimal  instruction  followed  by  working  through  exercises)  is  more  effective  than  unguided 
exploration,  behavioural  modelling,  computerized  tutorials,  or  classroom  instruction  alone. 
However,  research  also  shows  that  constructivist  techniques  that  combine  guided  exploration  and 
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behavioural  modelling  can  be  a  potent  combination  for  teaching  digital  skills  (Goodwin,  2006). 
Future  research  in  the  area  of  computer-based  instruction  may  also  need  to  consider  the  impact  of 
digital  skill  on  knowledge/skill  training. 

Building  on  the  requirement  to  have  adequate  digital  skills  for  computer-based  instruction, 
individuals  must  also  have  an  acceptable  level  of  comfort  with  technology.  Cannon-Bowers  and 
Bowers  (2008)  noted  that  negative  emotions  and  anxiety  associated  with  computer  or  technology 
use  can  severely  interfere  with  learning.  This  can  be  an  issue  when  using  new  or  complicated 
technology,  especially  when  immediate  assistance  with  technology  may  not  be  available  (e.g.,  as 
may  occur  with  distance  education  -  see  Capuzzi  &  Simon,  2007). 

2.5.5  Emerging  Research  Themes  and  Questions  -  Delivery  Media 

The  use  of  emerging  delivery  media  offers  strong  promise  of  promoting  learning  and  retention. 
These  technologies  range  from  very  simple  (e.g.,  single-task  activities  such  as  typing  tutors)  to 
complex  forms  of  simulation  (e.g.,  collaborative  simulations). 

Classroom  aids  can  be  simple  paper  and  pen  tools  that  help  to  engage  trainees  and  explicitly 
construct  mental  models.  Attention  to  the  physical  and  physiological  aspects  of  the  learning 
environment  was  seen  in  the  literature  review  to  have  impact  on  learning. 

Distance  learning  can  involve  different  approaches  to  the  use  of  technology.  Geographically 
distributed  synchronous,  or  asynchronous  courses  have  been  found  to  result  in  accelerated 
learning  in  both  education  and  industry;  though  this  may  be  most  relevant  in  certain  types  of 
learning,  and  for  certain  types  of  individuals  with  high  motivation  and  discipline  for  learning.  The 
advantages  of  saving  time  (e.g.,  no  travel  required,  less  teaching  time  required  with  canned 
instruction)  and  financial  resources  (e.g.,  cost  of  travel,  cost  of  instructors)  which  could  reduce 
costs  of  training  programs  may  be  of  particular  interest  to  the  CF. 

Computer-based  instruction  was  prominent  in  the  literature  reviewed.  Although  there  was  some 
evidence  that  computer-assisted  instruction  had  positive  impacts  on  learning,  variance  in  the 
range  of  complexity  of  computer-based  instruction  makes  it  difficult  to  draw  any  strong 
conclusions  about  its  general  effectiveness.  Flowever,  it  is  clear  that  using  intelligent  tutoring 
systems  has  very  strong  appeal  for  military  systems,  because  they  offer  a  range  of  time  and  cost 
saving  potential.  Given  the  potential  importance  of  computer-assisted  instruction,  it  will  be 
necessary  for  future  research  and  development  efforts  to  target  strong  empirical  knowledge  about 
what  kinds  of  computer-based  instruction  are  the  most  effective  for  a  given  range  of  tasks,  and 
how  these  systems  can  promote  more  efficient  and  effective  learning. 

Although  varied  in  range  and  scope,  there  is  also  strong  agreement  in  the  literature  that  the  use  of 
simulation  can  promote  better  learning.  At  the  simplest  possible  level,  for  a  system  that  uses  some 
forms  of  simulation  to  train  its  personnel,  the  key  issue  is  whether  simulation  training  actually 
does  what  it  is  intended  to  do.  For  example,  does  a  simulator  intended  to  train  skills  with  small 
arms  actually  promote  the  intended  learning  outcomes  (e.g.,  shooting  effectiveness)?  Even  if 
simulation  can  be  shown  to  be  effective,  though,  it  would  also  be  ideal  to  show  that  the 
effectiveness  of  training  is  significantly  different  from  conventional  classroom/demonstration 
training.  The  impact  of  simulation-based  training  could  also  be  explored  from  the  perspective  of 
the  level  of  transfer  that  can  be  shown  from  simulation-based  performance  to  real-world 
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performance  on  a  live  firing  range.  In  this  sense,  it  is  clear  that  performance  in  a  simulator  may  be 
the  result  of  actual  learning,  but  that  if  this  learning  cannot  be  “carried  over”  to  real-world 
environments;  it  may  be  of  limited  value. 

Understanding  what  aspects  of  simulation  are  most  critical  to  promoting  accelerated  learning 
seems  an  important  research  issue. 

The  literature  review  showed  a  number  of  possible  research  questions,  as  shown  in  Table  6. 


Table  6  Findings  and  Research  Questions-  Delivery  media 


Research  Finding 

Delivery  Media  -  Classroom  Aids  and  Environment 

Receptive,  behavioural,  guided  discovery,  exploration 

Problem-solving  focused  training  increases  learning  transfer  (Schaab 
&  Dressel,  2001) 

Lecture-based  instruction  is  found  to  be  less  favourable  by  many 
students  (Schaab  &  Dressel,  2001)  and  ineffective  in  terms  of 
accelerating  learning  (Meier,  2000) 

What  training  approaches  can  help  to 
accelerate  learning? 

Promoting  visualization/development  of  mental  models  promotes 
learning 

Novices  tend  to  have  mental  models  reflective  of  surface  features 
while  experts  have  more  abstract  mental  models  that  prove  more 
effective  in  performance  (Schumacher  &  Czerwinski,  1992;  cited  in 
Andrews  &  Fitzgerald,  2010). 

Visual  learning  techniques  tend  to  be  more  effective  than  verbal 
learning  techniques  regardless  of  the  preferred  learning  style  (Pashler 
et  al.,  2008). 

How  can  the  emergence  of  mental  models  be 
supported  by  training  techniques? 

Do  rich  mental  models  promote  accelerated 
learning? 

Are  mental  models  equally  relevant  to 
improving  training  effectiveness  in  varying 
skill  domains? 

Specific  classroom-aids 

“Hip  pocket”  training  -  impromptu  training,  short  15-20  minute 
lessons  between  the  “cracks”  (e.g.,  compass  training  during  a  water 
break  while  on  a  hike). 

Concept  maps  (mind-mapping),  flow  diagrams,  music,  “serious” 
games 

What  aids  are  most  conducive  to  learning? 

Do  these  largely  untested  training  aids  (e.g., 
hip  pocket,  concept  maps)  promote 
accelerated  learning? 

Positive  learning  culture  can  foster  effective  learning  (see  Blume  et 
al.,  2010;  Meier,  2000) 

Learning  transfer  is  significantly  impacted  by  the  transfer  climate 
(Blume  et  al.,  2010). 

All  5  senses  should  be  stimulated  in  order  to  achieve  holistic  learning 
(Meier,  2000;  TLC,  2010).  Overstimulation  or  unpleasant  stimulation 
should  be  avoided  (Erickson,  1991). 

What  aspects  of  the  training  environment 
most  influence  learning  effectiveness? 

Games  provide  novel  and/or  motivating  methods  of  learning 
(Erickson,  1991;  Cannon-Bowers  &  Bowers,  2008). 

What  technology  has  been  used  to  provide 
effective  gaming-based  training  that  has  led  to 
accelerated  learning? 

What  gaming-based  technologies  could  be 
implemented  to  accelerate  learning  within  the 
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Research  Finding 


CF? 

Delivery  Media  -  Distance  Learning 

Optimal  distance  learning  may  depend  on  type  of  skill/knowledge, 
resources,  and  additional  responsibilities  of  the  instructor  and 
student. 

What  skills  are  most  conductive  to  the  use  of 
distance  education? 

Do  skills  taught  via  distance  education  show 
similar  rates  of  decay? 

Delivery  Media  -  Simulation-Based  Instruction 

Students  who  are  exposed  to  and  participate  in  immersive  training 
learn  faster  and  more  effectively  (see  Meier,  2000). 

Simulations  can  be  used  to  create  realistic,  near  transfer  type  training 
programs  which  is  one  method  used  to  accelerate  learning  (see 
Cannon-Bowers  &  Bowers,  2008;  Zipperer  et  al.,  2003) 

Simulations  enable  time  compression  required  for  accelerated 
learning  (Hoffman,  Feltovich,  Fiore,  Klein,  &  Ziebell,  2009) 

How  much  fidelity  is  enough? 

What  aspects  of  fidelity  are  most  critical  to 
enabling  accelerated  learning  (e.g.,  contextual 
embeddedness,  tailoring  to  the  individual)? 
What  trade-offs  can  be  made  between 
simulation-based  instruction  and  actual 
practical  tasks  (e.g.,  flight  time,  range  time, 
sea  time)? 

Does  simulation-based  training  accelerate 
learning? 

How  can  simulation-based  training  be  used  to 
tailor  training  to  the  needs  of  the  individual? 

Do  all  types  of  learners  benefit  equally  from 
simulation-based  training? 

Does  simulation-based  training  facilitate 
better  transfer  than  conventional  training  to 
real-world  contexts? 

Technology  can  be  implemented  to  facilitate  training  of  geographically 
distributed  students  and  instructors  (see  Cannon-Bowers  &  Bowers, 
2008;  Capuzzi  Simon,  2007) 

In  team  environments,  can  simulated-based 
instruction  models  substitute  for  team 
members? 

How  can  simulation-based  training  be  used  to 
tailor  training  to  groups  and  teams? 

Can  simulation-based  training  accelerate 
learning  at  the  team  level? 

Delivery  Media  -  Computer-Based  Instruction 

Computer-based  instruction  (ranging  from  very  simple  to  very 
complex  tutoring  systems)  can  promote  higher  levels  of  learning.  For 
example,  web-presented  information  significantly  increased  test 
scores  compared  with  scores  for  control  group  (lecture  style) 
(Cannon-Bowers  &  Bowers,  2008). 

Can  computer-based  training  accelerate 
learning? 

Can  computer-based  training  facilitate  better 
levels  of  transfer  to  work  performance? 

What  forms  of  computer-based  instruction  are 
most  likely  to  help  accelerate  learning? 

How  does  computer-based  training  relate  to 
transfer  to  real-world  contexts? 

Do  intelligent  tutoring  systems  work  better  if 
they  are  tailored  to  the  needs  of  the 
individual? 

Are  there  means  to  accelerate  learning 
through  capitalizing  on  informal  learning  (vs. 
formal)  using  computer-based  and 
technology-based  information? 
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2.6  Delivery  Methods 

A  range  of  delivery  methods  are  noted  within  the  literature.  A  particularly  relevant  delivery 
method  noted  in  the  literature  is  collaborative  learning.  Delivery  methods  related  to  logistic 
aspects  of  planning  and  implementation  of  training  programs  such  as  scheduling,  feedback  and 
testing,  adaptive  instruction,  and  additional  context  and  content  factors  also  emerged. 

2.6.1  Collaborative  Learning 

Whether  students  learn  individually  or  in  a  group  context  may  also  influence  learning  (and  by 
extension,  accelerated  learning).  A  common  assumption  is  that  collaborative  learning  can 
promote  better  learning.  Collaborative  learning  can  be  defined  as  approaches  that  require  joint 
intellectual  efforts  by  two  or  more  students.  Some  definitions  also  note  the  importance  of  the 
instructor  within  these  collaborative  efforts.  An  important  distinction  in  the  literature  is  between 
team  training  and  collaborative  training  (Salas  &  Cannon-Bowers,  2001).  Salas  and  Cannon- 
Bowers  argue  that  team  training  is  relevant  to  a  team  task,  whereas  collaborative  learning 
involves  group  training,  but  not  necessarily  to  perform  a  task  as  a  team.  This  report  uses  the  term 
“group”  to  cover  both  of  these  areas.  Whatever  the  unit  of  analysis,  Meier  (2000,  p.viii) 
succinctly  argues  that  “collaboration  among  learners  enhances  learning”.  This  section  explores 
the  empirical  evidence  in  support  of  this  assertion. 

There  is  very  good  agreement  about  the  power  of  collaborative  learning.  Group  training  can 
promote  better  learning  because  peer  feedback  can  support  learning  performance,  as  well  as 
vicarious  learning  (Salas  &  Cannon-Bowers,  2001).  A  review  by  Salas  and  Cannon-Bowers 
(2001)  cites  research  by  Shebiliske,  Regian,  Arthur  and  Jordan  (1992)  showing  that  collaborative 
protocols  can  cut  instructor  time  and  resources  by  50%,  as  well  as  reducing  the  need  for  “hands- 
on”  practice  because  of  observational  learning  that  occurred.  Other  research  by  Zipperer  et  al. 
(2003)  within  the  military  domain  showed  that  levels  of  trust  within  teams  contributed  to  higher 
levels  of  skill  attainment,  as  well  as  increased  retention  of  these  skills.  Similarly,  teams  with 
shared  mental  models  have  also  been  consistently  shown  to  perform  better.  Encouraging  shared 
mental  models  through  high  levels  of  communication  or  even  cross-training  can  help  soldiers 
understand  how  their  peers,  leaders,  and  joint  services  think,  what  they  might  need,  and  when 
they  need  it  so  information  and  assistance  can  be  pushed  at  the  proper  time  before  the  person  who 
needs  it  has  to  request  it  (Zipperer  et  al.,  2003).  This  would  involve  team  members  having  clear 
knowledge  about  team  procedures,  the  capabilities  of  other  team  members,  interdependencies 
among  sub-tasks,  and  which  knowledge  needs  to  be  shared. 

Obviously,  the  prevalence  of  a  range  of  social  networking  tools  also  attests  to  the  potential  power 
of  collaborative  learning.  Sites  such  as  Facebook,  Twitter,  and  Linkedln  are  examples  of  ways  to 
support  and  promote  collaborative  learning  as  well  as  connectedness  in  general.  Whatever  the 
form,  then,  collaborative  learning  can  facilitate  learning  processes  and  promote  training 
effectiveness 

2.6.2  Scheduling 

The  scheduling  of  practice  and  training  is  a  critical  influence  on  learning  and  skill  acquisition. 
For  training,  for  example,  the  duration  (length  of  a  lesson  or  length  of  training  program), 
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frequency  (spacing  of  lessons),  and  the  rate  (speed  of  moving  through  content)  of  training  are  all 
important  factors  noted  in  the  literature.  Although  there  is  recognition  of  the  importance  of  these 
factors,  optimal  course  duration  and  frequency  of  lessons  or  retraining  is  highly  dependent  on  the 
type  and  complexity  of  the  task  (e.g.,  Komell  &  Bjork,  2008  and  Roher  &  Taylor,  2006;  both 
cited  in  Andrews  &  Fitzgerald,  2010)  and  the  length  of  time  material  that  must  be  retained  (e.g., 
Roher  &  Pashler,  2007;  cited  in  Andrews  &  Fitzgerald,  2010;  see  also  Ginzburg  &  Dar-El,  2000; 
Goodwin,  2006). 

The  importance  of  practice  when  working  to  acquire  a  new  skill  is  well  established.  Flowever,  a 
review  by  Salas  and  Cannon-Bowers  (2001)  laments  that  precise  conceptualization  of  practice 
and  exactly  how  it  relates  to  training  outcomes  has  been  overlooked  or  misunderstood.  They 
argue  that  practice  needs  to  be  conceptualized  as  something  more  than  simple  repetition.  Practice 
or  retraining  is  argued  to  be  of  particular  importance  when  tasks  are  infrequent  (Cannon-Bowers 
&  Bowers,  2008).  Research  has  identified  a  rule  of  thumb  to  be  used  in  assessing  optimal  spacing 
of  practice.  For  example,  Rohrer  &  Pashler  (2007;  cited  in  Andrews  &  Fitzgerald,  2010)  proposed 
that  optimal  intervals  between  practice  (and  refresher  training)  are  10-20%  of  the  retention 
interval  (duration  between  the  time  a  skill  is  learned  or  practiced  and  tested  or  performed)  (Lance, 
Parisi,  Bennett,  Teachout,  Harville,  &  Welles,  1998). 

There  appears  to  be  little  agreement  in  the  literature  about  the  impact  of  massed  vs.  spaced 
practice.  In  their  review,  Andrews  and  Fitzgerald  (2010)  identified  research  that  demonstrated 
that  spaced  practice  was  more  effective  than  massed  practice  in  learning  complex,  cognitive  skills 
Flowever,  little  research  was  found  related  to  procedural  skills.  Citing  earlier  research,  a  review 
by  Rowatt  and  Shlechter  (1993)  reported  that  there  was  little  difference  in  retention  as  the  result 
of  massed  vs.  spaced  practice.  However,  research  by  Wells  and  Hagman  (1983;  cited  in  Rowatt 
and  Shlechter.  1993)  argued  that  spaced  practice  was  optimal  in  3  specific  conditions: 

•  Dangerous  tasks  where  fatigue  would  present  risks  to  safety 

•  When  students  are  not  well  motivated 

•  For  high  ability  students,  who  might  over  respond  to  massed  practice  and  become 
fatigued. 

Even  after  training  ends,  a  schedule  that  promotes  active  participation  and  hard  work  should 
continue  in  order  to  maintain  skills  and  to  further  reinforce  and  develop  learning.  Andrews  and 
Fitzgerald  (2010)  identified  skill  “stretching”  as  an  important  way  to  promote  learning.  This 
involves  tailoring  to  the  individual  and  presenting  them  with  progressively  more  difficult 
challenges,  such  as  the  provision  of  tough  or  rare  cases.  This  will  help  them  to  identify  patterns  of 
responses  and  respond  to  them.  Thinking  through  these  cases  can  provide  opportunities  to 
implement  a  newly  learned  skill.  New  challenges  incoiporated  in  training  can  keep  students  “on 
their  toes”  and  maintain  their  interest  which  is  a  founding  component  of  accelerated  learning. 

Zipperer  et  al.  (2003)  explored  procedural  skills  and  these  researchers  argue  that  mass  training  of 
procedural  skills  (e.g.,  hours  of  repetition)  can  be  beneficial,  and  in  some  cases  may  be  required 
prior  to  expanding  or  building  on  that  skill.  They  also  emphasize  the  importance  of  mastering  the 
basics  before  adding  challenges  to  training,  as  performance  can  suffer  if  the  student  is  not 
proficient  with  the  basic  elements  of  the  skills  before  additional  levels  of  complexity  are  added 
(Zipperer  et  al.,  2003).  A  recommendation  noted  in  the  literature  is  to  allocate  additional  time  to 
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initial  learning  (particularly  with  respect  to  procedural  skills)  and  have  subsequent  practice  or 
retraining  sessions  shorter. 

The  scheduling  of  training  is  also  a  critical  influence  on  learning.  The  available  literature 
addressed  issues  related  to  time  compressed  training  (e.g.,  Andrews  &  Fitzgerald,  2010),  as  well 
as  just-in-time  training,  as  noted  by  Salas  and  Cannon-Bowers  (2001).  This  form  of  training  is 
scheduled  to  occur  just  before  it  is  required.  The  shortened  interval  between  skill  acquisition  and 
skill  usage  is  argued  to  be  a  benefit  of  this  form  of  training.  Unfortunately,  little  description  and 
information  was  available  in  the  literature  reviewed.  However,  scheduling  of  practice  and  training 
is  an  important  theme  noted  in  the  literature. 

2.6.3  Feedback  and  Testing 

Feedback  is  critical  to  learning  and  to  accelerated  learning.  Testing  is  one  way  that  students 
receive  feedback  about  their  skill  levels  relative  to  the  learning  goal.  Researchers  advocate  that 
“learners’  memory  for  information  or  procedures  can  be  directly  enhanced  through  testing” 
(Roedinger  and  Karpicke;  cited  in  Pashler  et  al.,  2008,  p.l  17).  For  example,  a  study  by  McDaniel, 
Roediger,  and  McDermott  (2007;  cited  in  Andrews  &  Fitzgerald,  2010)  showed  performance 
advantages  after  using  a  study-test  model  compared  with  a  study-study  model.  Testing  soon  after 
material  is  learned  seemed  to  enhance  learning  and  maximize  retention,  especially  when 
accompanied  by  corrective  feedback.  Retention  research  also  shows  that  students  who  were  tested 
(and  provided  feedback  on  their  degree  of  success)  during  training  performed  better  at  retention 
than  those  who  received  only  presentation  training  (Bryant  &  Angel,  2000;  Goodwin,  2006). 
Testing  may  simulate  the  actual  performance  environment  and/or  stimulate  the  development  of 
more  effective  strategies  by  students. 

Training  feedback  can  be  provided  by  any  number  of  sources.  Trainers  or  instructors  are  typically 
the  primary  source  of  feedback.  This  section  focuses  on  the  type  and  timing  of  feedback  and 
feedback  provided  from  peers  and  instructors.  Effective  feedback  must  provide  insight  into 
correcting  errors  and  information  relevant  to  improving  the  organization  of  knowledge  pertaining 
to  the  skill  or  task.  A  distinction  can  be  made  between  effective  feedback  that  provides  direction 
on  how  to  improve  performance  and  feedback  designed  for  encouragement  (e.g.,  motivational 
feedback).  The  former  can  assist  in  increasing  individual  capability  while  the  latter  may  improve 
personal  commitment.  Both  may  be  important  for  improving  performance;  however,  feedback 
specifically  discussed  within  accelerated  learning  literature  focuses  on  improving  performance 
capability.  Nonetheless,  the  importance  of  continued  motivation  of  the  student  is  evident 
throughout  the  literature  relevant  to  learning. 

Other  students  can  provide  critical  forms  of  feedback  that  promote  learning  through  shared 
problem  solving,  pointing  out  different  perspectives,  and  providing  corrective  criticism.  Using 
peer  feedback  (also  called  peer  tutoring)  during  learning  and  practice  is  noted  as  being  helpful  to 
learning  in  the  available  literature  and  has  been  shown  to  aid  learning.  Importantly,  research  by 
Schaab  &  Dressel  (2001)  suggests  that  peer  feedback  has  been  demonstrated  as  a  preferred 
method  of  learning  for  the  majority  of  soldiers.  Peer  discussion  and  collaboration  during  learning 
and  practice  has  been  shown  to  provide  students  with  necessary  information  to  progress  through  a 
task  and  create  knowledge.  A  study  at  Stanford  University  demonstrated  that  peer  feedback  was 
more  effective  than  several  other  factors  (i.e.,  reduced  class  size,  lengthened  instruction  time,  or 
individual  computer-based  instruction)  at  improving  math  and  reading  skills  (Levine,  Glass,  & 
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Meister,  1987;  cited  in  Meier,  2000).  Peer  feedback  can  be  encouraged  during  learning  by  pairing 
savvy  students  with  less  experienced,  novice  students  (Leibrecht,  Wampler,  Goodwin  &  Dyer,  in 
preparation;  cited  in  Schaab  &  Dressel,  2001).  Pairs  or  groups  of  students  can  collaborate, 
providing  feedback  to  one  another  as  an  effective  form  of  learning  because  it  fosters  information 
exchange  within  a  social  context  and  allows  students  to  share  multiple  perspectives  (Cannon- 
Bowers  &  Bowers,  2008).  As  such,  other  students  can  be  an  important  source  of  feedback. 

The  timing  of  feedback  is  also  a  critical  factor.  The  majority  of  studies  suggest  that  corrective 
feedback  during  learning  or  immediately  after  performance  may  be  more  beneficial  than  delayed 
feedback.  Immediate  feedback  or  help  during  problem  solving  has  been  found  to  be  beneficial 
(Lajoie,  2003  cited  in  Andrews  &  Fitzgerald,  2010).  Semb,  Ellis,  and  Araujo  (1992)  provide 
empirical  evidence  that  feedback  while  learning  (in  the  form  of  tutoring)  develops  knowledge  and 
leads  to  improved  learning  and  retention. 

In  general,  meaningful  and  timely  feedback  has  been  identified  by  a  number  of  researchers  as 
contributing  significantly  to  learning  (Meier,  2000;  Andrews  &  Fitzgerald,  2010).  What  this  looks 
like  in  practical  application  may  vary  significantly  based  on  the  type  of  skill  or  knowledge  being 
learned,  students’  preferred  style  of  learning,  and  resources  (e.g.,  time,  instructors, 
methodologies).  Whatever  the  form,  feedback  is  a  critical  part  of  learning  as  it  provides  advice  for 
improvements  to  performance  capabilities. 

The  type  of  feedback  provided  during  training  sessions  can  impact  later  retention.  When  feedback 
contains  information  about  the  magnitude  and  direction  of  performance  errors  the  learner  is 
directed  toward  correction,  which  can  lead  to  better  retention.  This  type  of  feedback  is  distinct 
from  motivational  feedback  which  though  useful,  does  not  provide  any  direct  performance 
suggestions.  Careful  attention  must  be  paid,  however,  to  the  frequency  of  feedback  as  students 
can  become  dependent  on  it.  Feedback  should  be  informative,  but  not  become  frequent  enough  to 
be  a  part  of  the  students’  mental  representation  of  the  task  (Bryant  &  Angel,  2000).  A  study  by 
Rowatt  and  Shlechter  (1993)  showed  that  corrective  performance  feedback  can  improve  soldier 
armour  skill  retention  as  long  as  soldiers  do  not  becoming  reliant  on  the  feedback. 

Similar  to  findings  related  to  feedback  described  within  the  accelerated  learning  literature, 
feedback  in  the  retention  literature  can  take  the  form  of  tests  when  accompanied  with  corrective 
criticism.  Retention  research  concurs  that  students  who  were  tested  (and  provided  feedback  on 
their  degree  of  success)  during  training  performed  better  at  retention  than  those  who  received 
only  presentation  training  (Bryant  &  Angel,  2000;  Goodwin,  2006).  Testing  may  simulate  the 
actual  performance  environment  and/or  stimulate  the  development  of  more  effective  strategies  by 
students. 

Feedback  should  be  incoiporated  into  training  programs  in  order  to  achieve  a  high  level  of 
retention.  Both  the  amount  of  feedback  and  feedback  frequency  require  optimization.  Further 
investigation  of  what  factors  affect  feedback  effectiveness  is  required  for  the  specific  training  in 
question. 

A  critical  issue  is  the  relationship  between  accelerated  learning  and  retention.  Even  if  learning  can 
successfully  be  accelerated,  a  critical  question  is  how  well  the  knowledge  and  skills  learned  will 
be  maintained  over  time.  Critics  suggest  that  retention  of  “speedy”  learning  would  suffer 
compared  to  retention  in  traditional  learning  (see  Capuzzi  Simon,  2007).  Flowever,  research  by 
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Bailey  (1989;  cited  in  Ginzburg  &  Dar-El,  2000)  suggests  that  the  rate  of  learning  is  not  related  to 
learning  decay  (or  forgetting)  which  is  a  function  of  the  amount  learned  and  the  break  duration. 


2.6.4  Adaptive  instruction 

In  traditional  training,  the  learning  rate  is  controlled  by  the  instructor.  Typical  classroom  learning 
consists  of  an  instructor  giving  a  presentation  or  lecture  and  material  is  covered  as  quickly  or  as 
slowly  as  the  instructor  decides. 

The  available  literature  is  clear  on  the  importance  of  adaptive  instruction.  This  can  be  defined 
simply  as  tailoring  to  ensure  some  match  between  how  the  learning  process  is  designed  and 
structured  and  the  needs  of  the  student.  There  is  some  evidence  in  the  literature  that  learning  can 
be  increased  by  providing  students  with  control  over  tasks  performed  or  by  providing  the  student 
with  control  over  the  pace  of  learning  (Cannon-Bowers  &  Bowers,  2008). 

The  introduction  of  technology  presents  new  options  for  customizing  training  to  accommodate 
various  rates  of  learning.  For  example,  blended  online  and  in-class  courses  are  becoming 
prominent  in  post-secondary  institutions.  Students  often  choose  the  type  of  course  they  would  like 
to  take,  which  gives  them  greater  power  in  controlling  how  the  information  is  presented  (e.g.,  on¬ 
line  courses  are  often  over  the  Internet  using  many  visual  resources)  and  the  pace  at  which  they 
learn  (e.g.,  in  online  courses,  the  pace  of  learning  is  often  controlled  by  the  learner)  (Capuzzi 
Simon,  2007).  Providing  this  choice  may  lead  to  greater  learning  if  students  feel  they  have  control 
and  responsibility  for  their  own  learning,  as  well  as  choosing  a  learning  pace  that  best  suits  their 
unique  learning  style  and  abilities.  Indeed,  research  has  shown  that  students  in  accelerated 
programs  can  achieve  final  grades  similar  or  better  than  those  of  college  students  learning  the 
same  material  (Capuzzi  Simon,  2007). 

Adaptive  instruction  is  relevant  both  to  the  original  design  of  the  course,  as  well  as  to  the  process 
that  unfolds  as  the  student  is  taught.  Intelligent  tutoring  systems,  for  example,  can  be  designed  to 
be  responsive  to  errors  that  the  student  makes  and  to  identify  problematic  themes  that  the  student 
does  not  seem  to  understand.  Targeted  remediation,  then,  is  a  critical  part  of  adaptive  instruction. 

Overall,  then,  there  is  good  agreement  that  adapting  instruction  to  match  the  needs  of  students 
will  help  promote  better  learning  outcomes. 

2.6.5  Context  and  Content 

Our  review  of  the  literature  showed  a  range  of  other  relevant  topics  that  are  specific  to  the  context 
or  content  of  training,  but  which  do  not  easily  fall  into  discrete  categories.  These  topics  are 
explored  within  this  section. 

2. 6.5.1  Training  Context 

The  importance  of  context  in  learning  is  a  persistent  theme  noted  during  the  literature  review.  As 
Meier  (2000,  p.  xviii)  argues  “the  best  learning  comes  from  doing  the  work  itself  in  a  continual 
process  of  “real-world”  immersion,  feedback,  reflection,  evaluation  and  re-immersion”.  This  is 
true  at  the  level  of  the  individual  student,  as  well  as  in  thinking  about  the  meaning  of  training 
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within  an  organizational  system.  Training  is  no  longer  seen  as  an  isolated  event  that  occurs 
outside  of  the  application  context,  but  “...organizations  have  shifted  their  views  about  training 
from  a  separate,  stand-alone  event  to  a  fully  integrated,  strategic  component  of  the  organization” 
(Salas  &  Cannon-Bowers,  2001,  p.  472).  Baldwin  et  al.  (2009)  also  noted  that  there  is  a  need  to 
more  explicitly  address  the  social  environment  and  organizational  context  of  training  activities 
and  also  a  need  to  categorize,  investigate,  and  report  contextual  variables  that  may  influence 
transfer  (Baldwin  et  al.,  2009). 

Research  is  clear  about  the  importance  of  contextual  learning,  and  suggests  that  “we  gain  the 
most  by  actually  doing  the  activity  that  we  are  being  trained  to  do”  (TLC,  2010,  p.2).  Training 
should  closely  reflect  the  actual  “in-the-field”  or  “on-the-job”  tasks  (Andrews  &  Fitzgerald,  2010; 
Meier,  2000;  TLC,  2010).  Achieving  good  results  in  performing  real-life  tasks  can  develop 
competence  and  confidence  essential  for  good  performance.  Zipperer  et  al.  (2003)  recommend 
realistic  hands-on  experience  for  soldier  training.  According  to  the  authors,  students  should  be 
immersed  in  realistic,  challenging  tactical  environments  and  exercises  should  include  live  fire, 
live  explosives,  heavy  loads,  medical  treatment,  evacuation  drills,  and  fast-paced  combat 
operations. 

Cannon-Bowers  and  Bowers  (2008)  further  argue  that  experiential  learning  is  vital  to  effective 
learning.  Anchored  instruction,  defined  as  instruction  that  is  based  in  a  meaningful  context, 
provides  students  with  experiences  of  how  a  concept  may  be  applied  which  leads  to  new  learning 
and  promotes  the  development  of  mental  models  and  integration  of  learned  information. 

Another  assertion  in  the  literature  is  that  learning  is  more  effective  when  learners  and  students  are 
provided  with  deeper  information  about  the  underlying  problem.  This  may  depend  on  whether  the 
goals  of  learning  are  related  to  “why”  or  “how”  and  “what”.  In  some  cases,  the  goal  of 
training/education  may  be  to  elucidate  the  deeper  principles  associated  with  an  issue.  In  other 
cases,  the  goal  is  simply  to  teach  students  how  to  perform  a  specific  and  isolated  skill.  Some 
researchers  have  argued  that  “training  must  provide  increasingly  detailed  knowledge  procedures 
and  principles,  in  context,  with  progressive  refinement  as  expertise  develops”  (Wulfeck,  2008;  as 
cited  in  Andrews  &  Fitzgerald,  2010,  p.6).  The  assertion  that  students  will  learn  better  if  taught 
the  foundational  principles  of  a  problem  rather  than  the  specifics  with  no  principles  or 
background  is  a  consistent  pattern  in  the  existing  literature  (e.g.,  Meier,  2000). 

2. 6. 5.2  Training  Content 

Various  aspects  of  training  content  are  noted  in  the  literature.  These  include  cross-training, 
overlearning  and  active  versus  passive  learning. 

Cross-training  is  one  prominent  form  of  training  noted  in  the  available  literature.  Cross-training 
is  designed  to  help  build  shared  team  mental  models.  Cross  training  requires  team  members  to 
understand  the  entire  team  function,  and  the  role  that  one’s  particular  tasks  plays  within  and 
between  those  of  other  team  members.  One  way  of  doing  this  is  by  having  team  members  swap 
roles  temporarily.  Volpe  et  al  (1996;  as  cited  in  Adams,  Webb,  Angel,  &  Bryant,  2003)  found  that 
cross  training  amongst  two-person  air  combat  crews  led  to  better  individual  performance,  more 
effective  team  communication,  and  improved  teamwork.  Further,  naval  three-person  teams  that 
received  cross  training  were  found  to  have  no  decrement  in  speed  or  accuracy  in  the  case  of 
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unexpected  role  substitution  (McCann  et  al.,  2000;  as  cited  in  Adams  et  al.,  2003);  teams  that  did 
not  receive  cross  training  did  see  a  significant  decrease  in  performance. 

Over-learning  is  another  technique  noted  in  the  accelerated  learning  literature.  Overlearning  is 
defined  as  training  and  practice  “beyond  the  attainment  of  criterion-level  performance”  (Lance  et 
al.,  1998.  p.  105;  see  also  Andrews  &  Fitzgerald,  2010;  Zipperer  et  al.,  2003).  That  is,  practicing  a 
task  not  only  to  the  point  of  proficiency,  but  over  and  beyond  that  point  until  the  task  seems  to 
come  automatically  and  students  could  perform  it  “with  their  eyes  closed”.  Over-learning  is 
essentially  considerable  repetition  of  a  task  during  training  and  is  often  used  in  the  acquisition  of 
physical  and  procedural  skills.  A  goal  of  over-learning  a  procedural  task  is  to  instil  muscle 
memory,  making  a  task  automatic  (Zipperer  et  al.,  2003).  Simple  repetition,  however,  is  not 
enough  to  effectively  master  a  skill.  Thought  and  reflection  should  supplement  and  precede 
repetitive  drills  (Zipperer  et  al.,  2003),  and  thus  must  be  considered  in  respects  to  planning  and 
time  allocation  for  training.  A  combination  of  repetition  and  visualization  may  be  as  effective  as 
an  overlearning  approach.  However,  it  is  also  important  to  note  recent  work  that  casts  some  doubt 
on  the  effectiveness  of  overlearning  (e.g.,  Andrews  &  Fitzgerald,  2010).  Nonetheless, 
overlearning  is  a  commonly  cited  training  technique  cited  in  both  the  learning  and  retention 
literature. 

Active  vs.  passive  learning  -  Another  factor  argued  to  impact  on  learning  and  on  accelerated 
learning  is  whether  the  learning  is  active  or  passive.  Active  learning  can  be  defined  as  engaging 
the  student  and  requiring  them  to  apply  effort  in  order  to  acquire  and  practice  a  skill  or 
knowledge.  Active  learning  is  often  associated  with  high  amounts  of  motivation,  and  more  learner 
involvement  (e.g.,  researching  information  themselves).  Passive  learning  requires  less  active 
involvement  on  the  part  of  the  study,  and  they  are  assumed  to  absorb  the  information  that  is  being 
presented  to  them  such  as  in  lecture -style  learning.  Active  learning  is  argued  to  promote  better 
learning.  Meier  (2000),  for  example,  argued  that  learning  requires  active  participation  and  hard 
work.  The  argument  that  learning  is  facilitated  by  active  motivation  and  participation  of  the 
learner  is  also  evident  in  the  term  “total  learner  involvement”  (Meier,  2000),  and  seems  to  be 
related  to  the  term  “experiential  learning”  (Cannon-Bowers  &  Bowers,  2008). 

There  is  good  evidence  showing  the  benefits  of  active  rather  than  passive  learning.  For  example, 
research  by  Healy,  Ericsson,  and  Bourne  ( 1 990)  showed  simply  reading  an  association  (whether 
multiplication  or  word  based)  was  not  as  effective  a  learning  process  as  having  subjects  generate 
or  recollect  (e.g.,  calculate)  the  proper  responses  during  initial  training.  This  “generation  effect”, 
involving  the  subjects  as  active  players  in  generating  responses,  resulted  in  a  much  higher 
retention  rate  when  assessed  weeks  and  months  later.  This  effect  was  demonstrated  through 
simple  numerical  associations.  Tasks  requiring  participants  to  perform  the  mental  calculation 
themselves  (verifying  numbers  and  generating  answers)  showed  higher  retention  rates  than  when 
the  learner  was  more  of  a  passive  observer  (e.g.,  reading  answers).  Thus,  the  researchers  provide 
evidence  that  when  the  student  is  actively  engaged  through  mental  calculation,  retention  is 
increased.  Active  participation  is  key  to  improving  learning  and  retention. 

There  is  some  evidence  in  the  available  literature  that  more  active  learning  can  promote  better 
learning  outcomes.  Research  by  Schaab  and  Dressel,  2001;  cited  in  Goodwin,  2006)  involved 
military  intelligence  officers  trained  using  either  traditional  techniques  (guided  demonstration)  or 
more  active  constructivist  techniques  (e.g.,  working  in  groups  on  practical  exercises  -  guided 
exploration).  Results  showed  that  although  both  groups  did  well  on  the  final  exam,  the  guided 
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exploration  group  showed  better  performance  on  novel  practical  exercises  and  reported  lower 
levels  of  cognitive  load.  This  suggests  that  active  learning  techniques  may  promote  better 
learning. 

New  Age  Techniques  -  A  review  by  Swets  and  Bjork  (1990)  evaluated  a  range  of  “new  age” 
techniques  for  training  conducted  by  the  Army  Research  Institute  (ARl).  The  National  Research 
Council  (NRC)  was  commissioned  to  assess  extraordinary  techniques  developed  outside  typical 
mainstream  research  in  behavioural  sciences.  A  committee  was  formed  through  the  NRC  called 
the  Commission  on  Techniques  for  the  Enhancement  of  Human  Performance  and  took 
responsibility  for  conducting  the  research.  They  audited  unique  training  techniques  that  were 
accompanied  by  strong  claims  of  effectiveness. 

Topics  specifically  of  interest  to  the  ARI  included  learning  efficiency,  improving  motor  skills, 
altering  mental  states,  stress  reduction,  group  cohesion  and  interpersonal  group  processes  (e.g., 
the  impact  of  group  cohesion  on  individual  and  group  performance)  and  parapsychological 
processes  (e.g.,  mental  influence  on  remote  machines).  The  ARl  was  interested  in  realizing 
possibilities  of  learning  while  sleeping,  or  accelerating  learning  based  on  packaged  programs  that 
include  guided  imagery,  mental  practice,  visual  concentration,  and  biofeedback.  Further, 
additional  concepts  were  explored  that  could  serve  to  optimize  performance  by  altering  a 
student’s  mental  state  by  means  of  self-induced  hypnotism,  meditation,  focused  concentration  and 
integration  of  activity  in  the  brain’s  hemispheres. 

The  NRC  committee  found  no  evidence  of  positive  learning  during  sleep,  for  the  usefulness  of  a 
specific  accelerated  learning  program  (SALT),  and  no  support  for  parapsychological  approaches 
to  training.  They  concluded  that  mental  practice  or  rehearsal  and  learning  was  well  supported,  but 
that  increasing  visual  skills  through  training  was  not,  and  evidence  was  lacking  for  the  use  of 
biofeedback  to  improve  motor  skills.  Stress  was  argued  to  impact  on  training  effectiveness,  and 
the  committee  found  that  neurolinguistic  programming  (NLP)  had  promise,  but  had  not  been 
shown  to  be  effective.  They  also  concluded  that  the  impact  of  group  cohesion  on  learning  was 
poorly  understood. 

V  isualization  and  mental  rehearsal  -  Other  reviews  focused  on  the  importance  of  being  able  to 
visualize  a  problem  as  an  important  part  of  learning.  Visualization  is  argued  to  promote  learning 
because  it  promotes  a  student’s  active  role  in  mentally  “organizing”  information  required  to 
perform  a  skill  or  task.  Some  researchers  suggest  that  visual  learning  techniques  tend  to  be  more 
effective  than  verbal  learning  techniques  regardless  of  a  student’s  preferred  learning  style 
(Pashler  et  al.,  2008).  The  emergence  of  a  mental  model  is  often  described  as  the  logical 
culmination  of  visualization  processes  and  the  accumulation  of  an  integrated  knowledge  structure. 
Encouraging  students  to  form  mental  models  is  another  strategy  commonly  used  to  promote 
higher  levels  of  learning.  Novices  become  increasingly  good  in  a  task  by  developing  more 
organized  and  abstract  mental  models.  Guided  imagery  can  also  lead  students  through  mentally 
performing  a  skill  or  task  that  can  speed  up  learning  and  improve  performance.  Mental  images 
created  by  the  student  and  have  been  shown  to  increase  recall  (TLC,  2010). 

Visualization  has  been  frequently  used  as  an  important  technique  in  mastering  skills  within 
military  contexts  (Zipperer  et  al.,  2003).  Military  training  subject  matter  experts  (SMEs)  agreed 
that  cultivating  realistic  scenario-based  imagery  of  a  task  is  an  essential  first  step  to  skill  mastery, 
particularly  in  procedural  tasks  conducted  by  dismounted  combat  troops.  To  aid  this  technique, 
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live,  video,  auditory,  and  model-based  or  simulation-based  demonstrations  could  provide 
necessary  information  on  which  visualization  can  be  based  (see  Erickson,  1991;  Fu  et  al.,  2009; 
Zipperer,  et  al.,  2003). 

A  technique  called  symbolic  mental  rehearsal  (SMR)  has  also  been  shown  to  improve  learning  of 
computer  skills.  Research  conducted  by  Davis  and  Yi  (2004;  cited  in  Goodwin,  2006)  required 
participants  to  use  this  mental  imaging  technique  to  parse  the  steps  of  a  task  into  a  spreadsheet, 
assign  labels  to  them,  and  to  mentally  rehearse  the  order  of  these  labels.  Results  showed  that 
SMR  improved  learning,  even  with  a  number  of  other  variables  (related  to  instructors,  age, 
computer  experience)  controlled. 

A  specific  learning  technique  relevant  to  the  emergence  of  mental  models  is  problem-based 
learning.  Collard  et  al.  (2009)  conducted  a  study  using  problem  based  learning  (PBL)  techniques 
that  focused  on  building  expertise  by  developing  knowledge  structures.  PBL  was  developed  to 
assist  students  in  developing  knowledge  constructs  into  organized,  effective  mental  models.  The 
purpose  of  PBL  is  to  improve  cognitive  processes,  particularly  problem  solving  skills,  and  build 
stronger  knowledge  networks  that  ease  acquisition  and  integration  of  new  information  and 
retrieval  of  stored  information. 

Collard  et  al.,  (2009)  investigated  the  factual  knowledge  and  the  strategic  skills  (i.e.,  reasoning)  of 
medical  students  enrolled  in  a  PBL-based  curriculum.  The  study  showed  that  year  3  students 
performed  better  on  factual  knowledge  measures  compared  with  upper  year  students  (year  4-6). 
Year  3  students,  as  expected,  performed  worse  than  upper  year  (5-6)  students  on  the  reasoning 
test.  For  the  lower-year  students,  factual  knowledge  and  reasoning  was  correlated.  This  may 
indicate  that  their  mental  models  (or  knowledge  networks)  used  in  reasoning  tasks  are  closely 
related  to  factual  knowledge.  Upper  year  students  did  not  show  this  correlation  which  suggests 
that  these  students,  having  more  experience  and  knowledge  of  the  field,  may  have  developed 
more  complex  and  abstract  mental  models  that  are  more  effective  for  problem  solving  and 
reasoning  tasks.  The  main  educational  difference  between  these  two  groups  was  hours  of  hands- 
on  experience.  Year  3  students  had  none,  upper  year  students  had  from  4-16  months  (halftime,  or 
a  mixture  of  halftime  and  fulltime).  Though  it  was  not  the  focus  of  this  study,  this  finding 
provided  evidence  that  may  support  practical  hands-on  application  of  knowledge  as  means  of 
developing  abstract,  expert  mental  models. 

As  a  whole,  then,  our  review  of  the  literature  suggests  that  there  are  a  range  of  contextual  and 
content-related  issues  likely  to  impact  on  learning. 

2.6.6  Emerging  Research  Themes  and  Questions  -  Delivery  Methods 

This  chapter  shows  that  many  different  delivery  methods  are  likely  to  impact  on  learning  and 
training  effectiveness.  These  include  collaborative  learning,  scheduling,  feedback  and  testing, 
adaptive  instruction  and  issues  related  to  context  and  content  within  which  training  occurs. 

Many  military  tasks  require  collaboration  and  teamwork.  Evidence  shows  that  learning  with 
others  (e.g.,  sharing  of  information,  developing  shared  mental  models)  can  promote  positive 
learning  outcomes.  Increased  use  of  social  networking  tools  may  also  promote  new  types  of 
collaboration  that  are  not  yet  well  understood,  nor  researched. 
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Scheduling  focuses  on  issues  of  timing,  duration,  and  spacing  of  initial  learning  and  subsequent 
practice  that  determines  the  frequency  and  intensity  of  which  trainees  are  expected  to  learn. 
Scheduling  lays  out  when  and  how  learning  and  testing  should  occur.  Although  there  is  some 
evidence  in  the  literature  about  the  impacts  of  varying  training  schedules  (e.g.,  just-in-time,  time 
compressed),  there  was  little  agreement  about  the  best  way  to  promote  a  high  level  of  proficiency 
in  a  reduced  amount  of  time. 

Feedback  is  noted  in  the  literature  as  an  important  influence  on  learning.  The  level  of  detail, 
frequency  (e.g.,  continuous  intermittent),  and  process  (e.g.,  timing  vs.  delayed,  source  of 
feedback)  of  providing  feedback  can  all  influence  learning.  Understanding  these  components  as 
they  relate  to  promoting  positive  learning  outcomes  requires  more  research,  and  it  may  be 
beneficial  to  look  at  the  specific  training  in  question.  Although  there  is  some  discussion  of  the 
role  of  both  instructors  and  peers  in  feedback,  there  is  little  sense  in  the  available  literature  that 
the  potential  power  of  feedback  from  peers  has  been  grasped.  As  training  often  occurs  within  a 
collaborative  environment,  the  issue  of  feedback  (and  particularly  the  nature  of  collaborative 
feedback)  may  be  worth  exploring  in  more  detail. 

There  is  some  agreement  in  the  literature  about  the  importance  of  tailoring  training  approaches  to 
the  needs  of  students.  Adaptive  instruction  may  be  one  method  of  increasing  their  motivation  or 
providing  students  with  a  sense  of  control  over  their  learning  which  (for  these  reasons  or  others) 
leads  to  accelerated  learning. 

Issues  of  context  and  content  are  also  critical  issues  to  thinking  about  delivery  methods.  These 
relate  to  the  efficacy  of  principle-based  learning  that  focuses  on  “why”  learning  needs  to  occur,  as 
opposed  to  the  simple  provision  of  information.  The  potential  impacts  of  overlearning  and 
different  forms  of  training  (e.g.,  cross-training)  also  need  to  be  better  understood  within  the 
unique  military  environment. 

There  is  a  rich  set  of  potential  issues  related  to  delivery  methods  that  would  be  possible  to  explore 
in  future  R  &  D  efforts.  Relevant  research  findings  as  well  as  potential  research  and  development 
opportunities  are  shown  in  Table  7. 


Table  7  Findings  and  Research  Questions-  Delivery  methods 


Delivery  Methods  -  Collaborative  Learning 

Collaborative  tools  can  promote  better 
learning 

Do  collaborative  tools  promote  learning? 

What  collaborative  technologies  show  the  most 
promise  of  promoting  accelerated  learning? 

What  collaborative  technologies  might  be  used  in 
training  to  promote  accelerated  learning  (e.g. 

Facebook,  Twitter,  SMS)? 

How  do  distributed  teams  learn  most  effectively? 

Collaborative  training  can  promote  learning. 
Collaboration  among  students  can  provide 
different  perspectives  and  enrich  the  learning 
experience  (see  Schaab  &  Dressel,  2001). 

What  types  of  skills  are  most  conducive  to 
collaborative  benefits  during  training? 
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Delivery  Methods-Scheduling 

Practice  -  Need  to  constantly  “stretch”  skills 
to  have  accelerated  learning  -  need  to  have 
increasing  levels  of  difficulty 

Spaced  practice  was  more  effective  than 
massed  practice  in  learning  complex, 
cognitive  skills  (Andrews  &  Fitzgerald,  2010). 
Procedural  skills  can  benefit  from  mass 
training  (e.g.,  hours  of  repetition)  (Zipperer  et 
al„  2003) 

Flow  should  practice  be  scheduled  (spaced  or  massed) 
for  varying  types  of  skills? 

What  is  the  relative  effectiveness  of  different  training 
schedules? 

What  kinds  of  “stretching”  of  skills  are  optimal  (e.g., 
complexity,  rates,  duration,  unique  challenges).  What 
are  the  ideal  rates  of  “stretching”? 

What  training  technologies  are  most  helpful  in 
providing  practice  that  “stretches”  skills  sets,  can  be 
tailored  to  the  performance  of  the  individual? 

Training  design  -  Evidence  supports  front 
loading  mass-instruction  (high  frequency)  for 
procedural  skills  (see  Zipperer  et  al.,  2003) 

Flow  should  training  be  designed  to  accelerate  learning 
within  specific  skill  areas? 

Delivery  Methods  -  Feedback  and  Testing 

Feedback  -  Immediate  feedback  during 
training  /performance  is  most  effective 
(Lajoie,  2003  cited  in  Andrews  &  Fitzgerald, 
2010;  Semb,  Ellis,  &  Aarujo,  1992) 

Feedback  can  be  provided  by  the  instructor 
or  peers  (i.e.,  tutoring)  (see  Schaab  & 

Dressel,  2001 ;  Zipperer  et  al.,  2003) 

What  are  the  optimal  rates  of  feedback? 

What  are  the  optimal  rates  of  testing? 

Does  feedback  help  to  accelerate  learning,  and  if  so, 
how? 

Does  real-time  feedback  to  trainers  of  student 
understanding  (e.g.  use  of  clickers  in  the  classroom) 
help  to  accelerate  learning? 

Do  instructor  and  system  feedback  have  the  same 
effects? 

Delivery  Methods  -  Adaptive  Instruction 

Learning  should  be  tailored  to  individuals 

Personalized  training  is  more  interesting, 
easier,  increases  motivation  and 
consequently  increases  learning  (see 

Andrew  &  Fitzgerald,  2010;  Meier,  2000) 

Low-ability  students  benefit  from  highly- 
structured  learning  environments  and  high- 
ability  students  benefit  from  low-structured 
learning  environments  (Pashler  et  al.,  2008) 

What  types  of  adaptive  instruction  are  most  critical  to 
promoting  accelerated  learning  (e.g.,  rate  of 
presentation,  practice)? 

Dynamic  remediation  -  What  training  technologies 
provide  the  best  flexibility  for  promoting  tailoring  of 
instruction  (e.g.  computer-based  tutors)? 

Flow  can  technology  be  designed  to  maximize 
responsiveness  to  the  student? 

Instructors  -  Matching  supervisory  styles  with 
the  capabilities  and  commitment  of  the 
student  can  result  in  more  effective  training 
(Hersey  &  Blanchard,  1977) 

What  instructor  behaviours  facilitate  accelerated 
learning? 

Flow  does  the  level  of  instructor  training  and  expertise 
influence  learning?  What  aspects  of  their  expertise  are 
most  critical  to  maximal  learning? 

Delivery  Methods  -  Context  and  Content 

Contextual  embeddedness  improves  learning 

Can  overlearning  help  to  accelerate  learning? 
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(Meier,  2000) 

How  much  stress  is  pest? 

Is  cross-training  a  good  method  of  promoting  learning 
and  retention? 

What  forms  of  contextual  learning  are  most 
advantageous? 

What  technologies  best  support  contextual 
embeddedness? 

Using  principles  (why  training  is  necessary) 
create  foundational  knowledge  that  improves 
learning  (Meier,  2000) 

Does  presentation  of  background  and  principles 
promote  accelerated  learning? 

What  are  the  time/efficiency  trade-offs  for  accelerated 
learning? 

2.7  Skill  and  Knowledge  Retention 

Previous  sections  addressed  factors  associated  with  accelerated  learning  and  transfer.  This  chapter 
samples  the  literature  concerning  retention  of  skills  and  knowledge. 

2.7.1  Introduction  to  Retention 

Retention  can  be  defined  as  the  maintenance  or  sustainment  of  learned  behaviours  without 
practice  (Schendel  &  Hagman,  1991;  cited  in  Rowatt  &  Shlechter,  1993).  Others  have  defined 
retention  as  “the  degree  of  competence  to  which  an  acquired  skill  [or  knowledge]  is  retained 
through  the  passage  of  time”  (Ginzburg  &  Dar-El,  2000,  p.  327).  Another  definition  is  that  “Skill 
fading  (or  reduction  of  skill  retention)  is  the  measurable  decrement  in  performance  of  a  skill 
relative  to  a  criterion”  (Bryant  and  Angel,  2000,  p.  14). 

In  general,  it  is  clear  that  skills  generally  decay  with  the  passage  of  time  (Ginzburg  &  Dar-El, 
2000).  Further,  skill  loss  research  suggests  that  there  is  a  proportionately  large  degree  of  skill 
fading  after  relatively  moderate  amounts  of  time,  as  shown  in  Figure  2. 
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Performance 


Figure  2  Relationship  between  retention  performance  and  retention  interval  (Adams  et  al.,  2003) 

Retention  is  a  function  of  the  amount  learned  and  the  duration  between  task  performances.  This 
interval  between  learning  or  performance  and  a  subsequent  performance  is  called  the  retention 
inten’al  or  Skill  Retention  Interval  (SRI)  (e.g.,  Lance  et  al.,  1998).  This  interval  is  graphically 
represented  in  Figure  3. 


Time 


Retention  Interval 


Learning  /  Practice 


Skill  performance  (e.g.,  on-the- 
job  performance,  testing 
performance,) 


Figure  3  Skill  Retention  inten’al  (SRI) 


2.7.2  Retention-  Relationship  to  learning  factors 

Given  that  there  is  obviously  a  relationship  between  the  acquisition  of  knowledge  and  skills  and 
their  retention  over  time,  a  key  issue  is  the  exact  nature  of  this  relationship. 

Our  review  of  the  literature  offered  some  limited  insight  into  factors  that  influence  retention,  and 
showed  a  number  of  factors  that  are  likely  to  affect  retention  rates  after  learning.  These  factors 
include  student  and  instructor  attributes,  the  media  through  which  training  is  delivered,  the 
methods  used  to  conduct  training  (e.g.,  schedule  of  practice). 

Our  review  of  the  literature  showed  some  overlap  with  some  notable  findings  in  the  learning 
literature.  For  example,  both  literatures  address  some  common  student  attributes,  such  as 
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cognitive  ability  and  relevant  background  knowledge.  Although  cognitive  ability  has  been  shown 
to  increase  the  rate  at  which  information  is  acquired,  it  does  not  seem  to  affect  the  rate  of  decay. 
However,  because  individuals  with  higher  ability  may  learn  at  a  faster  rate,  they  are  typically  able 
to  achieve  a  higher  overall  level  of  learning  within  the  same  training  duration  (Goodwin,  2006). 
This  heightened  level  of  original  learning,  then,  means  that  they  can  show  better  retention.  As 
skills  have  been  shown  to  fade  at  a  somewhat  consistent  rate  (independent  of  aptitude),  people 
with  higher  cognitive  abilities  will  retain  more  than  people  with  moderate  rates  of  initial  learning. 

Background  knowledge  is  also  cited  in  the  literature  as  a  potential  influence  on  skill  retention. 
This  has  been  shown  in  a  range  of  studies  reported  by  Goodwin  (2006).  This  finding  is 
attributable  to  the  fact  that  better  background  information  on  a  given  topic  would  facilitate  the 
emergence  of  a  more  organized  and  coherent  knowledge  structure  that  might  be  easier  to  retain 
over  time.  However,  this  area  is  particularly  underdeveloped  from  a  research  perspective  and 
seems  to  lack  strong  designs  and  controlled  experiments. 

Other  individual  factors  prominent  in  the  learning  literature,  such  as  self-efficacy,  motivation,  and 
positive  emotions  are  less  prominent  in  the  literature  relevant  to  retention.  However,  expertise  is 
argued  to  increase  retention  of  newly  formed  skills  (  Andrews  &  Fitzgerald,  2010). 

Although  there  are  many  forms  of  delivery  media  (e.g.,  classrooms,  distance  learning,  simulator- 
based  learning),  direct  comparisons  of  their  impacts  on  retention  were  not  found  in  the  literature. 
There  is  evidence  that  existing  literature  is  inadequately  developed  to  begin  to  answer  this 
question.  For  example,  a  review  by  Goodwin  (2006,  p.  8)  exploring  the  literature  relevant  to 
digital  skills  argues  that  the  retention  levels  associated  with  the  efficacy  of  specific  training 
approaches  (e.g.,  guided  vs.  unguided)  “is  still  a  relative  unknown”.  Moreover,  the  studies  that  do 
exist  do  not  extend  more  than  a  month,  and  rely  primarily  on  tests  occurring  soon  after  the  end  of 
training. 

Delivery  methods  have  clear  connections  with  skill  and  knowledge  retention  in  the  literature.  The 
schedule  and  nature  of  refresher  training  and  its  relationship  with  retention  are  notable  examples 
of  this  connection.  Retention  of  a  skill  can  be  maintained  if  retraining  or  practice  is  implemented 
at  an  optimal  time.  However,  many  factors  influence  this  relationship  and  would  affect  the 
decision  of  timing  and  implementation.  Ginzberg  and  Dar-El  (2000)  argue  that  although  it  is  clear 
that  skills  fade  over  time,  the  right  kind  of  retraining  can  return  skilled  performance  to  similar 
original  levels,  and  this  relearning  time  is  quicker  than  the  initial  training  though,  with  more 
passage  of  time  between  original  learning  and  relearning  periods,  relearning  will  take  more  time. 
They  also  note  a  concept  called  the  “warming  up”  phenomenon,  namely  that  the  most  critical 
moments  of  retraining  are  the  first  few  minutes.  Active  participation  is  another  training  technique 
that  has  been  shown  to  increase  learning  and  accelerate  training,  as  well  as  promoting  better 
levels  of  retention  and  performance  (see  Meier,  2010). 

The  retention  literature  supports  the  effectiveness  of  training  programs  that  are  spaced  according 
to  the  nature  of  the  task  and  individual  students  in  order  to  minimize  fatigue,  increase  motivation, 
and  ensure  a  proficient  level  of  learning  is  achieved.  Optimal  durations  between  training 
segments  were  not  specified  as  they  would  be  affected  by  the  particular  skills  and  students 
involved.  Unfortunately,  the  logistics  of  training  programs  often  dictate  the  duration  and 
frequency  of  training,  resulting  in  training  and  practice  being  massed  together  (e.g.,  longer  hours 
of  training  with  fewer  sessions).  This  is  particularly  true  in  the  military.  For  retention,  this  can 
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have  negative  effects.  It  is  problematic  “...that  conditions  that  make  performance  improve  rapidly 
during  instruction  or  training  such  as  blocking  or  temporal  massing  of  practice  can  fail  to  support 
long-term  retention  and  transfer.”  (Pashler  et  al.,  2008,  p.  117). 

A  higher  level  of  original  learning  has  been  related  to  higher  retention  performance.  As  a  review 
of  the  retention  literature  by  Rowatt  &  Shlechter,  1993,  p.  2)  states  “. .  .nearly  all  of  these  reviews 
have  indicated  that  this  variable  is  the  single  best  determinant  of  skill  retention  with  the 
relationship  between  original  learning  and  retention  remaining  highly  positive  and  stable  for  an 
indefinite  period  of  time”. 

The  most  common  means  of  reducing  the  retention  interval  is  through  the  provision  of  refresher 
training.  The  length  of  time  required  for  refresher  training  (or  relearning)  has  been  consistently 
found  to  be  shorter  than  the  original  training  period.  More  specifically,  studies  have  shown  that 
the  length  of  time  required  for  refresher  training  is  less  than  50%  of  the  time  required  for  the 
initial  training  (Rowatt  &  Shlechter,  1993).  However,  long  intervals  between  initial  training  and 
retraining  mean  that  relearning  will  require  more  time. 

There  is  evidence  in  the  literature  that  more  practice  does  not  always  mean  higher  retention. 
Shute  and  Gawlick  (1995;  cited  in  Bryant  &  Angel,  2000)  compared  a  full  practice  session  to  an 
abbreviated  practice  condition  (which  was  about  25%  of  the  full)  and  found  those  in  the 
abbreviated  condition  had  better  retention.  They  supposed  that  students  in  the  abbreviated 
condition  devoted  greater  effort  during  training  to  compensate  for  fewer  practice  trials.  Other 
students  received  half  abbreviated  training  and  half  full  training.  These  students  performed  the 
best  of  all,  suggesting  that  variable  practice  schedules  enhance  retention  (Bryant  &  Angel,  2000). 
This  finding  was  further  supported  by  Goodwin  (2006)  who  found  that  greater  interference  at 
time  of  learning  (i.e.,  a  random  versus  a  blocked  schedule)  produced  higher  levels  of  retention 
and  transfer. 

A  study  by  Lance  et  al.  (1998)  conducted  a  study  with  8  samples  of  U.S.  Air  Force  personnel 
(from  different  specialities)  investigating  the  relationship  between  retention  interval  and  task 
performance  in  terms  of  3  potential  moderators.  These  included  initial  skill  learning,  student 
aptitude,  and  the  level  of  task  difficulty.  Initial  skill  learning  was  assessed  based  on  the  students’ 
experience  levels.  Results  showed  that  the  longer  the  SRI,  the  poorer  the  task  performance; 
however,  the  SRI  did  not  account  for  most  of  the  variance  in  test  performance.  Specifically,  via  a 
regression  analysis,  they  concluded  that  only  “...about  10%  of  the  variance  in  task  performance 
was  accounted  for  by  the  length  of  SRI”  (p.  118).  This  might  have  been  because  of  restriction  of 
range,  as  there  were  few  long  SRIs  in  the  sample,  and  participants  were  more  homogeneous  (due 
to  being  selected)  than  was  ideal.  The  lack  of  experimental  control  (e.g.,  participants  might  have 
been  required  to  perform  similar  tasks  in  other  parts  of  their  jobs),  and  other  moderators  that  were 
unmeasured  might  have  lowered  the  strength  of  the  relationship.  Initial  skill  learning  and  student 
aptitude  did  not  significantly  moderate  the  retention  interval/task  performance  relationship. 
Unfortunately,  although  this  study  asks  important  questions,  the  scientific  quality  of  this  research 
is  problematic. 

The  type  of  task  being  performed  is  reported  to  influence  optimal  timing  for  retraining  or 
practice.  For  example,  spaced  trials  tend  to  result  in  less  skill  decay  for  verbal  tasks  (Goodwin, 
2006).  A  review  by  Goodwin  (2006)  found  little  within  the  literature  on  the  effects  of  different 
training  schedules  on  retention  of  digital  skills.  Training  for  a  particular  skill  requires 
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examination  of  the  literature  pertaining  to  that  skill,  and  /or  its  own  empirical  research  to 
investigate  retention  effects  as  there  seems  to  be  no  effective  “one  size  fits  all”  methodology  for 
scheduling.  However,  there  is  some  evidence  in  the  literature  that  some  conventional  measures  of 
aptitude  show  differential  retention  of  different  types  of  skills.  For  example,  Goodwin  (2006) 
cites  previous  research  as  showing  that  ability  scores  were  more  strongly  associated  with 
retention  of  performance  on  mental  tasks  than  “hands-on”  or  psychomotor  tasks.  This  suggests 
that  cognitive  abilities  may  interact  with  the  type  of  skill  in  question  to  influence  retention  of 
these  skills. 

Ginzburg  and  Dar-El  (2000)  used  a  partial  simulator  to  investigate  the  relationship  between 
relearning  and  skill  retention  duration  in  military  training  relevant  to  Electronic  Warfare. 
Participants  from  the  Israel  Defence  Force,  Electronic  Warfare  Unit  were  divided  into  four 
groups,  each  having  a  different  retention  interval  (retention  intervals  were  1  month,  2  months,  3 
months,  and  6  months).  The  specific  tasks  and  procedures  using  the  partial  simulator  were  not 
detailed;  however  the  task  was  described  as  operating  a  complex  electronic  system. 

Results  showed  that  procedural  skill  decay  occurred  faster  than  decay  of  psychomotor  skills 
(Ginzburg  &  Dar-El,  2000),  and  controlled  skills  decayed  faster  than  automatic  skills.  Findings 
indicated  that  operators’  level  of  performance  conformed  to  the  typical  retention  curve 
(performance  declined  as  the  retention  interval  increased).  Refresher  training  fully  restored  skill 
levels  in  the  one  month  and  two  month  conditions.  After  three  months,  however,  skills  were  not 
fully  restored  through  this  additional  refresher  training.  At  this  point,  more  training  was  required 
to  fully  restore  skill  proficiency.  The  authors  argue  that  for  this  specific  training  task,  two  months 
was  the  optimal  length  of  time  between  retraining  in  order  to  restore  operator  performance  to  the 
acceptable  level  of  proficiency. 

The  similarity  between  the  training  situation  and  the  transfer  situation  are  also  noted  in  the 
literature  as  a  critical  influence  on  levels  of  retention  (Rowatt  and  Shlechter,  1993).  When  the 
training  task  and  the  transfer  task  share  common  structural  elements,  students  are  likely  to  be 
show  higher  levels  of  transfer.  Obviously,  this  means  that  tailoring  the  training  environment  to 
the  nature  of  the  skills  to  be  achieved  is  critical.  However,  an  interesting  issue  noted  in  the 
literature  is  that  the  impact  of  functional  similarity  is  relative  rather  than  absolute.  Specifically, 
student  perceptions  about  functional  similarity  seem  to  matter  more  rather  than  objective  function 
similarity  (Druckman  and  Bjork,  1991;  cited  in  Rowatt  and  Shlechter  (1993).  This  is  also  in  line 
with  high  fidelity  offering  learning  advantages.  The  retention  environment  needs  to  provide  the 
perceptual  and  cognitive  cues  required  to  retrieve  the  learned  skill  from  memory,  otherwise 
performance  will  suffer  (Bryant  &  Angel,  2000).  A  simple  example  involves  typing  in  numerical 
data.  Healy  et  al,  (1990)  looked  at  numerical  data  entry  on  the  number  row  versus  using  a 
numerical  keypad.  They  found  that  skill  retention  (measured  as  speed  of  data  entry)  was  only 
maintained  when  using  the  same  type  of  data  entry.  Thus,  entering  a  sequence  of  numbers  on  the 
keypad  only  resulted  in  faster  entry  times  if  entered  on  the  keypad  during  retention  testing.  If 
first  entered  on  the  number  row,  and  then  on  the  keypad  then  significantly  less  of  a  retention 
effect  was  found  even  if  the  same  sequence  of  numbers  was  used.  They  concluded  that  “it  is 
crucial  to  make  sure  that  the  procedures  we  use  when  learning  the  information  are  reinstated  at 
the  time  we  need  to  recall  the  information”  (Healy  et  al.,  1990,  p.  97). 

Both  field  and  laboratory  retention  studies  have  been  conducted  to  understand  retention. 
Generally,  laboratory  studies  have  shown  greater  retention;  however,  critics  point  out  that  tasks 


DRDC  Toronto  CR  201 1-105 


41 


are  often  fabricated  and  may  not  reflect  realistic  work-related  tasks  (Lance  et  al.,  1998).  Research 
demonstrated  that  armour  skill  retention  is  a  function  of  several  instructional  and  task  factors. 
Specifically  related  to  near  transfer,  researchers  found  that  soldier  retention  of  armour  skills  can 
be  maximized  by  making  training  requirements  functionally  similar  to  on-the-job  performance 
requirements  (Rowatt  &  Shlechter,  1993).  Thus,  retention,  as  measured  through  task 
performance,  is  expected  to  be  better  when  training  more  closely  reflects  the  evaluated  task.  This 
is  further  supported  by  research  on  “leam-by-doing”  techniques  which  have  been  shown  to 
improve  learning  of  procedural  skills  (Healy,  1997;  cited  in  Bryant  &  Angel,  2000).  Applying 
this  more  generally  to  military  settings,  Hagman  &  Rose  (1983;  as  cited  in  Rowatt  &  Shlechter, 
1993)  noted  that  field  exercises  are  more  appropriate  for  promoting  task  acquisition  and  retention 
for  military  field  purposes.  In  general,  near  transfer  can  facilitate  retention.  Training  programs, 
practice,  and  refresher  training  should  focus  on  tasks  that  directly  reflect  on-the-job  performance. 

How  knowledge  is  organized  during  training  has  also  been  shown  to  impact  retention.  The  easier 
tasks  are  to  organize  mentally,  the  faster  the  tasks  are  learned  (Adams  et  al.,  2003).  This  applies 
to  learning  simple  data,  as  well  as  to  more  complex  procedural  tasks,  which  if  well  organized, 
show  longer  retention  levels.  Essentially,  simpler  tasks  should  be  more  easily  organized  than 
more  complex  tasks.  Additional  time  to  learn  and  organize  information  related  to  complex  tasks 
may  be  required  in  order  to  maintain  and  retain  the  skill.  Effective  organizational  strategies  can 
also  be  implemented  by  instructors.  Some  examples  include  reminding  learners  of  their  currently 
possessed  knowledge  and  how  that  knowledge  relates  to  the  new  knowledge,  making  repeated  use 
of  information  presented  from  different  perspectives,  and  encouraging  students  to  elaborate  on  the 
material  during  learning  (as  well  as  during  later  application).  When  appropriately  applied,  each  of 
these  strategies  has  led  to  improved  learner  retention  (Adams  et  al.,  2003).  In  terms  of  feedback, 
retention  research  states  that  students  who  were  tested  (and  provided  feedback  on  their  degree  of 
success)  during  training  performed  better  at  retention  than  those  who  received  only  presentation 
training  (Bryant  &  Angel,  2000;  Goodwin,  2006). 

As  a  whole,  then,  research  suggests  that  optimal  spacing  for  training  and  amount  of  training  and 
practice  should  be  developed  based  on  the  unique  characteristics  of  the  training  program  in 
question.  Randomly  spaced  training  and  compressed  training  schedules  (e.g.,  implementing 
abbreviated  versions  of  training  programs)  can  increase  retention.  However,  motivation  of  the 
students  and  other  such  factors  may  also  impact  retention  outcomes. 

There  is  some  consistency  in  the  factors  noted  to  influence  skill  acquisition  and  skill  retention.  As 
previously  mentioned,  the  learning  and  training  literature  suggests  that  techniques  such  as  over¬ 
learning  can  be  used  to  train  individuals  above  proficiency.  Similarly,  over-learning  has  also  been 
shown  to  enhance  retention  (Bryant  &  Angel,  2000;  Goodwin,  2006;  Lance  et  al.,  1998;  Rowatt 
&  Shlechter,  1993).  Research  by  Driskell,  Willis  and  Cooper  (1992;  cited  in  Salas  &  Cannon- 
Bowers,  2001)  showed  a  significant  relationship  between  overlearning  and  retention,  but  this 
relationship  was  moderated  by  the  level  of  overlearning,  length  of  retention  interval  and  the  type 
of  task.  It  is  believed  that  over-learning  enhances  the  strength  of  associations  between  stimulus 
and  response  as  well  as  encouraging  automaticity  (Schendel  &  Hagman,  1982;  cited  in  Lance  et 
al.,  1998).  Earlier  research  also  showed  that  overlearning  promotes  student  confidence  and 
reduces  stress  (Martens,  1974:  cited  in  Lance  et  al.,  1998).  Some  limits  to  the  benefits  of  over¬ 
learning  have  been  found.  Driskell,  Willis  and  Cooper  (1992;  cited  in  Bryant  &  Angel,  2000) 
conducted  a  meta-analysis  and  suggested  that  a  minimum  of  50%  over-learning  was  required  to 
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gain  any  positive  effects.  This  suggests  that  valuable  training  time  can  be  wasted  if  too  little 
over-learning  is  attempted. 

As  noted  earlier,  learning  research  also  argues  that  the  more  involvement  that  subjects  have  in 
active  participation  during  learning,  the  better  their  subsequent  performance.  This  pattern  also 
seems  to  apply  to  retention.  This  was  demonstrated  in  an  experiment  conducted  by  Healy  et  al. 
(1990)  which  showed  that  when  subjects  were  active  players  in  generating  responses  they  had  a 
much  higher  retention  rate  weeks  and  months  later. 

However,  the  factors  that  influence  skill  acquisition  are  not  necessarily  the  same  ones  that 
influence  skill  retention.  For  example,  one  specific  factor  that  shows  a  distinct  pattern  relates  to 
contextual  interference.  Although  some  researchers  have  argued  that  having  a  consistent,  blocked 
schedule  of  training  leads  to  faster  acquisition  of  learning  than  with  a  random  schedule, 
interference  has  the  opposite  effect  on  the  retention  of  learned  information.  Battig  (1979:  cited  in 
Goodwin,  2006,  p.  17)  proposed  an  “intra-task  interference  principle  of  memory”  namely,  that 
“greater  interference  at  the  time  of  learning  produces  higher  levels  of  subsequent  retention  and 
transfer”.  Goodwin  (2006)  cites  reports  showing  contextual  interference  effects  associated  with  a 
30-50%  gain  in  recall  performance. 

For  the  future  a  critical  requirement  is  to  understand  the  relationship  between  the  factors  that 
promote  learning  and  those  that  influence  retention. 

2.7.3  Models  of  Retention 

For  trainers  and  educators,  the  ideal  would  be  to  know  when  skills  that  have  been  acquired  need 
to  be  refreshed.  Unfortunately,  there  are  no  prescribed  standards  for  the  amount  of  training  or 
practice  required  to  sustain  skill  levels,  but  there  are  some  models  that  are  relevant. 

For  example,  Bryant  and  Angel  (2000)  discussed  a  number  of  quantitative  predictive  models 
related  to  retention.  The  most  promising  of  these  in  their  view  was  the  Army  Research  Institutes’ 
(AR1)  Users’  Decision  Aid  (UDA).  This  model  allows  trained  evaluators  to  answer  ten  simple 
task-related  questions  (e.g.,  the  number  of  steps,  execution  demands,  task  time  limits,  etc.)  and 
calculate  the  “retention  score”  for  that  task.  The  UDA  model  splits  the  complexity  of  a  task  into 
three  categories:  number  of  steps,  whether  the  sequence  of  the  steps  matters,  and  whether  or  not 
there  is  inherent  feedback  that  indicates  correct  task  performance.  Each  of  these  sub-factors  was 
found  to  have  effects  on  skill  retention.  Once  calculated,  retention  scores  can  then  be  looked  up 
on  a  prediction  table,  to  determine  the  frequency  with  which  refresher  training  should  be  applied. 
This  model  has  been  empirically  evaluated  within  a  limited  domain  and  found  to  be  an  effective 
tool  for  determining  refresher  training  frequency. 

Unfortunately,  this  model  seems  best  suited  to  relatively  simple  procedural  skills,  and  there  is 
little  evidence  in  the  literature  of  other  pragmatic  models  that  would  help  to  understand  the 
retention  of  more  complex  skills. 
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2.7.4  Techniques  for  Retention 


Our  review  of  the  literature  showed  a  number  of  tools  and  aids  that  might  help  to  accelerate 
learning.  These  tools,  of  course,  are  designed  to  instantiate  some  of  the  core  principles  of  learning 
described  earlier  in  this  report.  For  example,  concept  maps  (or  mind  maps)  are  intended  to 
promote  the  use  of  visualization.  Unfortunately,  the  scientific  validity  of  these  tools  remains 
unclear,  and  whether  they  are  likely  to  help  accelerate  learning  remains  unknown.  Having 
validated  tools  and  aids  shown  to  facilitate  learning  and  retention  could  be  an  important 
contribution. 

Mnemonics  are  also  noted  in  the  literature  as  likely  to  help  promote  retention  of  skills.  They  are 
defined  as  “strategies  employed  to  impose  meaning  and/or  organization  on  complex  materials  or 
skill  to  facilitate  learning  or  retention  (Wells  &  Hagman,  1991;  cited  in  Rowatt  and  Shlechter, 
1993,  p.  8).  Research  suggests  that  allowing  learners  to  develop  their  own  mnemonics  enhances 
retention  and  transfer  of  procedural  tasks  (Druckman  &  Bjork,  1991;  Wells  &  Hagman,  1991; 
cited  in  Rowatt  &  Shlechter,  1993).  Other  memory  aids  are  also  noted  in  the  literature  (e.g., 
Erickson,  1991)  as  tools  that  can  facilitate  the  movement  of  information  from  short-term  memory 
to  long-term  memory.  Memory  aids  often  combine  memory  strategies  with  visualization 
techniques.  Our  review  of  the  literature  showed  some  evidence  that  memory  strategies  may 
promote  better  levels  of  retention.  For  example,  a  study  looking  at  retention  found  that  soldiers 
better  retain  armour  skills  when  they  are  encouraged  to  develop  an  organizational  mental  model 
of  the  task  (Rowatt  &  Shlechter,  1993).  This  finding  is  in  line  with  other  researchers  who  have 
argued  that  connection  to  and  integration  of  new  information  with  other  knowledge  (see  Andrews 
&  Fitzgerald,  20 1 0)  will  promote  retention. 

In  general,  retention  seems  to  increase  through  techniques  that  require  deliberate  effort  on  the  part 
of  the  student.  How  training  programs  could  be  designed  to  promote  better  levels  of  retention  is 
an  important  goal  of  future  research. 

2.7.5  Emerging  Research  Themes  and  Questions  -  Skill  and  Knowledge 
Retention 

This  review  of  the  skill  retention  literature  suggests  a  number  of  key  gaps  in  current  knowledge 
and  research  about  skill  fading.  A  serious  gap  in  the  current  literature  is  the  lack  of  strong 
research  depicting  complex  tasks,  studied  over  a  prolonged  retention  interval.  Current  literature  is 
typically  constrained  to  relatively  simple  tasks  with  limited  retention  intervals.  It  seems  important 
for  future  research  to  promote  better  understanding  about  the  retention  of  higher  order  cognitive 
skills  (e.g.,  planning,  problem  solving  and  decision-making)  after  training,  and  to  explore  the 
retention  of  these  skills  over  longer  time  frames. 

Moreover,  given  the  need  to  know  how  often  to  schedule  refresher  training,  accurate  prediction  of 
retention  levels  for  specific  tasks  is  currently  lacking.  Although  the  U.S.  military  has  developed 
the  UDA  model  to  do  this,  the  status  of  this  model  in  recent  years  is  unclear,  and  there  is  little 
evidence  in  the  available  literature  that  it  is  currently  in  use.  Moreover,  as  noted  earlier  this  model 
does  not  appear  to  be  particularly  well-suited  to  complex  tasks. 
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Another  gap  within  the  current  body  of  knowledge  that  seems  particularly  relevant  to  training  in 
the  CF  is  the  retention  of  collective  skills.  Given  the  need  for  many  military  teams  to  perform 
skills  as  a  coherent  unit,  a  critical  issue  is  the  retention  of  these  skills.  Unfortunately,  this  issue 
was  reported  to  be  particularly  under-researched  in  a  previous  review  (Adams  et  al.,  2003),  and 
there  is  little  indication  that  more  current  research  has  worked  to  fill  this  void. 

This  review  shows  that  the  link  between  emerging  technologies  and  skill  retention  is  an  in  serious 
need  of  more  empirical  investigation.  There  is  a  clear  assumption  in  the  skill  retention  literature 
that  technology  could  be  used  to  help  offset  the  loss  of  skilled  performance.  For  example,  a 
review  of  the  literature  exploring  retention  of  armour  skills  (Rowatt  and  Shlechter,  1993,  p.  11) 
concludes  that  future  research  should  focus  on  “Examining  the  effectiveness  of  simulators  and 
other  automated  training  devices  in  augmenting  skill  retention”.  Given  the  increasing  use  of 
simulation  within  training  systems,  a  critical  question  is  the  extent  to  which  skills  trained  with 
simulators  are  subject  to  the  same  or  different  levels  of  decay.  The  retention  of  specific  types  of 
skill  (e.g.,  procedural  or  more  complex  cognitive  skills)  being  trained  would  also  be  critical  to 
examine  within  this  area  of  research.  Technology  might  be  helpful  with  working  to  design  and 
present  training  materials  in  ways  that  research  argues  could  offset  skill  decay.  For  example, 
given  that  mental  models  are  argued  to  promote  better  levels  of  retention,  designing  simulations 
with  the  specific  aim  of  building  more  complex  mental  models  would  be  an  obvious  target  of 
future  research. 

Within  this  area,  there  is  a  rich  set  of  potential  issues  that  would  be  possible  to  explore  in  future 
research  and  development  efforts.  Relevant  research  findings  and  possible  research  questions  that 
emerge  from  the  literature  are  shown  in  Table  8. 


Table  8  Findings  and  Research  Questions-  Skill  Retention 


Learning  Factors  and  Retention 

Individuals  with  higher  cognitive  ability  show  higher 
retention  rates  because  they  learn  more  (Rowatt  & 
Shlechter,  1993).  However,  rates  of  decay  are  similar  for 
all  ability  levels. 

Expertise  can  increase  the  retention  of  new  skills. 

Does  acceleration  of  learning  impact  negatively  on 
retention? 

What  is  the  relationship  between  learner  factors  and 
retention  factors? 

What  are  the  most  critical  student  factors  that  impact 
retention  of  skills? 

Training  design  -  Retention  is  better  when  training  more 
closely  reflects  the  evaluated  task. 

Retention  can  be  maximized  by  making  training 
requirements  functionally  similar  to  on-the-job 
performance  requirements  (Rowatt  &  Shlechter,  1993). 
Greater  interference  at  time  of  learning  (i.e.,  a  random 
versus  a  blocked  schedule)  produced  higher  levels  of 
retention  and  transfer  (Goodwin,  2006). 

Can  increase  contextual  information  (i.e.,  in  rich 
simulation  environments)  promote  higher  levels  of 
retention? 

Can  contextual  interference  while  training  facilitate  higher 
levels  of  retention? 

Refresher  training  -  Length  of  time  required  for  refresher 
training  is  less  than  50%  of  the  time  required  for  the  initial 
training. 

Variable  practice  schedules  enhance  retention  (Bryant  & 

What  is  the  best  trade-off  between  cost/risk  for  refresher 
training  in  a  range  of  tasks? 

Can  ensuring  testing  and  feedback  soon  after  training 
facilitate  improved  retention? 

DRDC  Toronto  CR  201 1-105 


45 


Angel,  2000). 

Retention  research  concurs  that  students  who  were 
tested  (and  provided  feedback  on  their  degree  of 
success)  during  training  performed  better  at  retention 
than  those  who  received  only  presentation  training 
(Bryant  &  Angel,  2000;  Goodwin,  2006). 

Level  of  original  learning  is  an  excellent  predictor  of 
retention  performance  (Rowatt  &  Shlechter,  1993) 

What  is  the  best  learning/retention  trade-off? 

Overlearning  -  Overlearning  can  promote  better  retention 
in  some  situations,  but  results  are  inconsistent.  However, 
research  has  shown  that  50%  over-learning  required  to 
achieve  positive  effects  for  some  skills  (Bryant  &  Angel, 
2000). 

For  what  tasks/skills  is  overlearning  effective? 

Does  overlearning  impact  individuals  similarly? 

Is  overlearning  cost  effective? 

Feedback  -  When  feedback  contains  information  about 
the  magnitude  and  direction  of  performance  errors  the 
learner  is  directed  toward  correction,  which  can  lead  to 
better  retention. 

Feedback  should  be  informative,  but  not  become  frequent 
enough  to  be  a  part  of  the  students’  mental  representation 
of  the  task  (Bryant  &  Angel,  2000). 

Can  the  content  of  feedback  be  tailored  (e.g.,  in 
automated  systems)  to  facilitate  better  levels  of  retention? 

Computer-assisted  training  can  promote  retention 

Does  computer-assisted  training  promote  skill  retention? 

If  so,  is  computer-based  training  effective  for  all  learners 
and  types  of  tasks? 

Simulation-based  training  can  promote  retention  - 
Technology  may  be  particularly  useful  in  circumstances 
where  time  resources  are  limited  and  brief  episodes  of 
skill  performance  can  provide  some  form  of  practice  that 
will  help  to  retain  competency  (Andrews  &  Fitzgerald, 

2010). 

Does  simulation-based  training  promote  skill  retention? 

If  so,  is  simulation-based  training  effective  for  all  learners 
and  types  of  tasks? 

Models  of  Skill  Retention 

A  number  of  models  have  been  developed  to  predict  skill 
retention  (Bryant  &  Angel,  2000). 

Retention  interval  -  Performance  declines  as  the 
retention  interval  is  increased  (Semb  et  al.,  1992;  Bryant 
and  Angel,  2000).  The  level  of  skill  degradation  depends 
on  the  type  of  skill 

Can  models  of  skills  retention  predict  skill  decay? 

Techniques  for  Retention 

Stronger,  more  abstract  and  developed  mental  models 
can  also  increase  retention  of  skills  (Rowatt  & 

Shlechter, 1993) 

Allowing  learners  to  develop  their  own  mnemonics 
enhances  retention  and  transfer  of  procedural  tasks 
(Druckman  &  Bjork,  1991;  Wells  &  Hagman,  1991;  cited  in 
Rowatt  &  Shlechter,  1993). 

What  retention  techniques  are  the  most  effective? 

Can  mental  models  improve  rates  of  retention? 

What  kinds  of  tools/aids  are  most  helpful  for  improving 
retention  rates?  Are  there  tools  equally  applicable  to  all 
tasks? 
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Discrete  procedural  tasks  (e.g.,  job  knowledge)  are  more  Can  a  predictive  model  depicting  retention  of  varying 
quickly  forgotten  than  more  complex  continuous  skills  types  of  skills  be  created? 

(Schaab  and  Dressel,  2001)  (e.g.,  cognitive  skills) 


2.8  Critical  Findings:  Discussion  and  Emerging  Themes 

This  review  included  26  articles  from  three  major  bodies  of  literature,  training,  learning  and 
retention.  Our  review  of  the  literature  suggests  that  there  are  many  influences  on  the  acquisition 
and  retention  of  knowledge  and  skills.  These  include  characteristics  of  the  student,  the  instructor, 
and  delivery  media  and  methods  used. 

Our  literature  review  showed  that  the  construct  of  accelerated  learning  is  still  at  a  relatively  early 
stage  of  development.  Indeed,  there  is  currently  little  that  truly  distinguishes  the  literature  on 
accelerated  learning  from  research  that  explores  learning  at  a  general  level.  The  only  discemable 
difference  is  that  the  accelerated  learning  and  generic  learning  literature  seem  to  use  distinct 
terminology  to  describe  similar  principles  of  learning.  Moreover,  the  accelerated  learning  concept 
is  used  by  different  parties  for  different  puiposes.  The  most  relevant  literature  relevant  to 
accelerated  learning  comes  from  the  U.S.  military  domain  (e.g.,  Andrews  &  Fitzgerald,  2010). 

This  work  does  provide  general  guidelines  for  accelerated  learning,  however  this  work  has  a  more 
constrained  scope  in  that  it  focused  on  the  emergence  of  expertise  within  complex  domains  and 
learning  principles  rather  than  how  learning  of  either  simple  or  complex  tasks  can  be  more 
quickly  (and  better)  facilitated. 

Unfortunately,  although  the  review  clearly  showed  many  different  concepts  as  being  relevant  to 
accelerated  learning  and  retention,  research  is  not  currently  developed  enough  to  provide  any 
strong  conclusions  about  relative  importance  of  these  concepts,  or  sets  of  factors,  relative  to  each 
other  (or  even  within  a  set).  Looking  at  the  many  different  approaches  to  training,  for  example, 
there  is  no  clear  answer  about  what  factors  are  truly  the  most  influential.  Meta-analytic  efforts  to 
compare  multiple  factors  do  not  appear  to  have  emerged  yet.  This  lack  of  clarity  is  the  result  of 
controlled  research  in  these  areas  simply  not  having  been  directed  specifically  at  the  issue  of 
promoting  accelerated  learning.  The  lack  of  systemization  in  the  literature,  and  the  inability  to 
understand  the  effect  of  a  variable  in  one  setting  on  its  performance  in  another  is  perhaps  the  key 
challenge  in  understanding  the  status  of  training  in  20 1 1 . 

Nonetheless,  there  was  some  agreement  in  the  available  literature  about  the  issues  and  factors  that 
influence  learning  and  retention.  Fourteen  key  concepts  affecting  learning  and  retention  were 
identified  from  the  literature  review,  as  follows. 

1 .  student  attributes, 

2.  instructor  attributes, 

3.  classroom  aids  and  environment, 
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4.  distance  learning, 

5.  simulator-based  instruction, 

6.  computer-based  instruction, 

7.  collaborative  learning 

8.  scheduling 

9.  feedback  and  testing, 

10.  adaptive  instruction, 

1 1 .  context  and  content, 

12.  learning  factors  and  retention, 

13.  models  of  retention,  and 

14.  techniques  for  retention. 

Table  9  provides  a  summary  of  some  of  the  relevant  research  findings  noted  in  the  literature  and 
sample  research  questions  that  could  be  used  to  address  some  of  the  current  gaps  with  each  of 
these  areas. 


Table  9  Summary  of  Findings  and  Research 


Student  Attributes 

Student  attributes  impact  on  learning 

What  student  attributes  are  most  critical  to  learning? 

What  is  the  relative  importance  of  student  attributes  relative 
to  other  sets  of  factors  (e.g.,  delivery  media)? 

Cognitive  ability  promotes  learning  (Salas  &  Cannon- 
Bowers,  2001) 

How  can  learning  be  optimized  for  individuals  with  low 
cognitive  abilities? 

High  motivation  to  learn  increases  learning  (Blume  et 
al.,  2010;  Meier,  2000).  Hands-on  experience  can 
increase  motivation. 

Understanding  relevance  of  training  can  increase 
motivation  to  learn  application  of  newly  learned  skills. 
(TLC,  2010;  Zipperer  et  al.,  2003). 

How  can  training  be  designed  to  maximize  student 
motivation? 

What  approaches  and  technologies  (e.g.  virtual  worlds)  best 
promote  immersion,  engagement,  social  presence,  how  best 
to  promote  “flow” 

Embodiment,  presentation  and  engagement  (physical 
representation  of  the  user,  e.g.,  fidelity  of  the  avatar) 

Learning  styles  may  influence  learning 

Should  learning  styles  be  further  investigated? 

Prior  knowledge  or  experience  can  improve  learning 
(e.g.,  computer  background  knowledge  has  an  impact 
on  training  and  retention  of  digital  skills  (Goodwin, 

2006). 

How  can  prior  knowledge  and  experience  be  brought  to  bear 
on  training  effectiveness? 

Level  of  comfort  with  technology  can  slow  or  increase 
learning  and  motivation  to  learn  when  using 
technological  tools  (Capuzzi  Simon,  2007;  Zipperer  et 
al.,  2003)Expertise  improves  learning  (Andrews  & 

How  does  experience  and  comfort  with  technology  impact 
on  learners  in  technology-based  training?  What 
compensatory  approaches  might  support  personnel  with  low 
levels  of  experience? 
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Fitzgerald,  2010) 

How  can  training  build  expertise  more  quickly? 

How  can  expertise  best  be  leveraged? 

Can  emerging  technologies  support  the  emergence  of 
expertise? 

Instructor  Attributes 

Instructor  attributes  (e.g.,  subject  matter  expertise, 
teaching  experience  and  teaching  style)  impact  on 
learning 

What  are  the  most  critical  aspects  of  instructor 
effectiveness? 

What  is  the  ideal  mix  of  subject  matter  expertise  and 
teaching  experience? 

What  makes  a  good  mentor? 

Delivery  Media  -  Classroom  Aids  and  Environment 

Receptive,  behavioural,  guided  discovery,  exploration 
Problem-solving  focused  training  increases  learning 
transfer  (Schaab  &  Dressel,  2001) 

Lecture-based  instruction  is  found  to  be  less  favourable 
by  many  students  (Schaab  &  Dressel,  2001)  and 
ineffective  in  terms  of  accelerating  learning  (Meier, 

2000) 

What  training  approaches  can  help  to  accelerate  learning? 

Promoting  visualization/development  of  mental  models 
promotes  learning 

Novices  tend  to  have  mental  models  reflective  of 
surface  features  while  experts  have  more  abstract 
mental  models  that  prove  more  effective  in 
performance  (Schumacher  &  Czerwinski,  1992;  cited  in 
Andrews  &  Fitzgerald,  2010). 

Visual  learning  techniques  tend  to  be  more  effective 
than  verbal  learning  techniques  regardless  of  the 
preferred  learning  style  (Pashler  et  al.,  2008). 

How  can  the  emergence  of  mental  models  be  supported  by 
training  techniques? 

Do  rich  mental  models  promote  accelerated  learning? 

Are  mental  models  equally  relevant  to  improving  training 
effectiveness  in  varying  skill  domains? 

Specific  classroom-aids 

“Hip  pocket”  training  -  impromptu  training,  short  15-20 
minute  lessons  between  the  “cracks”  (e.g.,  compass 
training  during  a  water  break  while  on  a  hike). 

Concept  maps  (mind-mapping),  flow  diagrams,  music, 
“serious”  games 

What  aids  are  most  conducive  to  learning? 

Do  these  largely  untested  training  aids  (e.g.,  hip  pocket, 
concept  maps)  promote  accelerated  learning? 

Positive  learning  culture  can  foster  effective  learning 
(see  Blume  et  al.,  2010;  Meier,  2000) 

Learning  transfer  is  significantly  impacted  by  the 
transfer  climate  (Blume  et  al.,  2010). 

All  5  senses  should  be  stimulated  in  order  to  achieve 
holistic  learning  (Meier,  2000;  TLC,  2010). 
Overstimulation  or  unpleasant  stimulation  should  be 
avoided  (Erickson,  1991). 

What  aspects  of  the  training  environment  most  influence 
learning  effectiveness? 

Games  provide  novel  and/or  motivating  methods  of 
learning  (Erickson,  1991;  Cannon-Bowers  &  Bowers, 

What  technology  has  been  used  to  provide  effective  gaming- 
based  training  that  has  led  to  accelerated  learning? 
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2008). 

What  gaming-based  technologies  could  be  implemented  to 
accelerate  learning  within  the  CF? 

Delivery  Media  -  Distance  Learning 

Optimal  distance  learning  may  depend  on  type  of 
skill/knowledge,  resources,  and  additional 
responsibilities  of  the  instructor  and  student. 

What  skills  are  most  conductive  to  the  use  of  distance 
education? 

Do  skills  taught  via  distance  education  show  similar  rates  of 
decay? 

Delivery  Media  -  Simulation-Based  Instruction 

Students  who  are  exposed  to  and  participate  in 
immersive  training  learn  faster  and  more  effectively 
(see  Meier,  2000). 

Simulations  can  be  used  to  create  realistic,  near 
transfer  type  training  programs  which  is  one  method 
used  to  accelerate  learning  (see  Cannon-Bowers  & 
Bowers,  2008;  Zipperer  et  al.,  2003) 

Simulations  enable  time  compression  required  for 
accelerated  learning  (Hoffman,  Feltovich,  Fiore,  Klein, 

&  Zeibell,  2009) 

How  much  fidelity  is  enough? 

What  aspects  of  fidelity  are  most  critical  to  enabling 
accelerated  learning  (e.g.,  contextual  embeddedness, 
tailoring  to  the  individual)? 

What  trade-offs  can  be  made  between  simulation-based 
instruction  and  actual  practical  tasks  (e.g.,  flight  time,  range 
time,  sea  time)? 

Does  simulation-based  training  accelerate  learning? 

How  can  simulation-based  training  be  used  to  tailor  training 
to  the  needs  of  the  individual? 

Do  all  types  of  learners  benefit  equally  from  simulation- 
based  training? 

Does  simulation-based  training  facilitate  better  transfer  than 
conventional  training  to  real-world  contexts? 

Technology  can  be  implemented  to  facilitate  training  of 
geographically  distributed  students  and  instructors  (see 
Cannon-Bowers  &  Bowers,  2008;  Capuzzi  Simon, 

2007) 

In  team  environments,  can  simulated-based  instruction 
models  substitute  for  team  members? 

How  can  simulation-based  training  be  used  to  tailor  training 
to  groups  and  teams? 

Can  simulation-based  training  accelerate  learning  at  the 
team  level? 

Delivery  Media  -  Computer-Based  Instruction 

Computer-based  instruction  (ranging  from  very  simple 
to  very  complex  tutoring  systems)  can  promote  higher 
levels  of  learning.  For  example,  web-presented 
information  significantly  increased  test  scores 
compared  with  scores  for  control  group  (lecture  style) 
(Cannon-Bowers  &  Bowers,  2008). 

Can  computer-based  training  accelerate  learning? 

Can  computer-based  training  facilitate  better  levels  of 
transfer  to  work  performance? 

What  forms  of  computer-based  instruction  are  most  likely  to 
help  accelerate  learning? 

How  does  computer-based  training  relate  to  transfer  to  real- 
world  contexts? 

Do  intelligent  tutoring  systems  work  better  if  they  are  tailored 
to  the  needs  of  the  individual? 

Are  there  means  to  accelerate  learning  through  capitalizing 
on  informal  learning  (vs.  formal)  using  computer-based  and 
technology-based  information? 
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Delivery  Methods  -  Collaborative  Learning 

Collaborative  tools  can  promote  better  learning 

Do  collaborative  tools  promote  learning? 

What  collaborative  technologies  show  the  most  promise  of 
promoting  accelerated  learning? 

What  collaborative  technologies  might  be  used  in  training 
to  promote  accelerated  learning  (e.g.  Facebook,  Twitter, 
and  SMS)? 

Flow  do  distributed  teams  learn  most  effectively? 

Collaborative  training  can  promote  learning. 
Collaboration  among  students  can  provide  different 
perspectives  and  enrich  the  learning  experience  (see 
Schaab  &  Dressel,  2001), 

What  types  of  skills  are  most  conducive  to  collaborative 
benefits  during  training? 

Delivery  Methods-Scheduling 

Practice  -  Need  to  constantly  “stretch”  skills  to  have 
accelerated  learning  -  need  to  have  increasing  levels 
of  difficulty 

Spaced  practice  was  more  effective  than  massed 
practice  in  learning  complex,  cognitive  skills  (Andrews 
&  Fitzgerald,  2010).  Procedural  skills  can  benefit  from 
mass  training  (e.g.,  hours  of  repetition)  (Zipperer  et 
al.,  2003) 

Flow  should  practice  be  scheduled  (spaced  or  massed)  for 
varying  types  of  skills? 

What  is  the  relative  effectiveness  of  different  training 
schedules? 

What  kinds  of  “stretching”  of  skills  are  optimal  (e.g., 
complexity,  rates,  duration,  unique  challenges).  What  are 
the  ideal  rates  of  “stretching”? 

What  training  technologies  are  most  helpful  in  providing 
practice  that  “stretches”  skills  sets,  can  be  tailored  to  the 
performance  of  the  individual? 

Training  design  -  Evidence  supports  front  loading 
mass-instruction  (high  frequency)  for  procedural  skills 
(see  Zipperer  et  al.,  2003) 

Flow  should  training  be  designed  to  accelerate  learning 
within  specific  skill  areas? 

Delivery  Methods  -  Feedback  and  Testing 

Feedback  -  Immediate  feedback  during  training 
/performance  is  most  effective  (Lajoie,  2003;  cited  in 
Andrews  &  Fitzgerald,  2010;  Semb  et  al.  1992) 

Feedback  can  be  provided  by  the  instructor  or  peers 
(i.e.,  tutoring)  (see  Schaab  &  Dressel,  2001;  Zipperer 
et  al.,  2003) 

What  are  the  optimal  rates  of  feedback? 

What  are  the  optimal  rates  of  testing? 

Does  feedback  help  to  accelerate  learning,  and  if  so,  how? 

Does  real-time  feedback  to  trainers  of  student 
understanding  (e.g.  use  of  clickers  in  the  classroom)  help 
to  accelerate  learning? 

Do  instructor  and  system  feedback  have  the  same  effects? 

Delivery  Methods  -  Adaptive  Instruction 


Learning  should  be  tailored  to  individuals 
Personalized  training  is  more  interesting,  easier, 
increases  motivation  and  consequently  increases 
learning  (see  Andrew  &  Fitzgerald,  2010;  Meier,  2000) 


What  types  of  adaptive  instruction  are  most  critical  to 
promoting  accelerated  learning  (e.g.,  rate  of  presentation, 
practice)? 

Dynamic  remediation  -  What  training  technologies  provide 
the  best  flexibility  for  promoting  tailoring  of  instruction  (e.g. 
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Low-ability  students  benefit  from  highly-structured 
learning  environments  and  high-ability  students 
benefit  from  low-structured  learning  environments 
(Pashleretal.,  2008) 

computer-based  tutors)? 

How  can  technology  be  designed  to  maximize 
responsiveness  to  the  student? 

How  can  neuroscience  best  be  applied  to  military  teams? 

Instructors  -  Matching  supervisory  styles  with  the 
capabilities  and  commitment  of  the  student  can  result 
in  more  effective  training  (Hersey  &  Blanchard,  1977) 

What  instructor  behaviours  facilitate  accelerated  learning? 
How  does  the  level  of  instructor  training  and  expertise 
influence  learning?  What  aspects  of  their  expertise  are 
most  critical  to  maximal  learning? 

Delivery  Methods  -  Context  and  Content 

Contextual  embeddedness  improves  learning  (Meier, 
2000) 

Can  overlearning  help  to  accelerate  learning? 

How  much  stress  is  pest? 

Is  cross-training  a  good  method  of  promoting  learning  and 
retention? 

What  forms  of  contextual  learning  are  most  advantageous? 
What  technologies  best  support  contextual 
embeddedness? 

Using  principles  (why  training  is  necessary)  create 
foundational  knowledge  that  improves  learning  (Meier, 
2000) 

Does  presentation  of  background  and  principles  promote 
accelerated  learning? 

What  are  the  time/efficiency  trade-offs  for  accelerated 
learning? 

Learning  Factors  and  Retention 

Individuals  with  higher  cognitive  ability  show  higher 
retention  rates  because  they  learn  more  (Rowatt  & 
Shlechter,  1993).  However,  rates  of  decay  are  similar 
for  all  ability  levels. 

Expertise  can  increase  the  retention  of  new  skills. 

Does  acceleration  of  learning  impact  negatively  on 
retention? 

What  is  the  relationship  between  learner  factors  and 
retention  factors? 

What  are  the  most  critical  student  factors  that  impact 
retention  of  skills? 

Training  design  -  Retention  is  better  when  training  more 
closely  reflects  the  evaluated  task. 

Retention  can  be  maximized  by  making  training 
requirements  functionally  similar  to  on-the-job 
performance  requirements  (Rowatt  &  Shlechter,  1993). 
Greater  interference  at  time  of  learning  (i.e.,  a  random 
versus  a  blocked  schedule)  produced  higher  levels  of 
retention  and  transfer  (Goodwin,  2006). 

Can  increased  contextual  information  (i.e.,  in  rich  simulation 
environments)  promote  higher  levels  of  retention? 

Can  contextual  interference  while  training  facilitate  higher 
levels  of  retention? 

Refresher  training  -  Length  of  time  required  for 
refresher  training  is  less  than  50%  of  the  time  required 
for  the  initial  training. 

Variable  practice  schedules  enhance  retention  (Bryant 
&  Angel,  2000). 

What  is  the  best  trade-off  between  cost/risk  for  refresher 
training  in  a  range  of  tasks? 

Can  ensuring  testing  and  feedback  soon  after  training 
facilitate  improved  retention? 
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Students  who  receive  both  feedback  and  testing 
perform  better  than  with  only  presentation  training 
(Bryant  &  Angel,  2000;  Goodwin,  2006). 

Level  of  original  learning  is  an  excellent  predictor  of 
retention  performance  (Rowatt  &  Shlechter,  1993) 

What  is  the  best  learning/retention  trade-off? 

Overlearning  -  Overlearning  can  promote  better 
retention  in  some  situations,  but  results  are 
inconsistent.  However,  research  has  shown  that  50% 
over-learning  required  to  positive  effects  for  some  skills 
(Bryant  &  Angel,  2000). 

For  what  tasks/skills  is  overlearning  effective? 

Does  overlearning  impact  individuals  similarly? 

Feedback  -  When  feedback  contains  information  about 
the  magnitude  and  direction  of  performance  errors  the 
learner  is  directed  toward  correction,  which  can  lead  to 
better  retention.  (Bryant  &  Angel,  2000). 

Can  the  content  of  feedback  be  tailored  (e.g.,  in  automated 
systems)  to  facilitate  better  levels  of  retention? 

Computer-assisted  training  can  promote  retention 

Does  computer-assisted  training  promote  skill  retention?  If 
so,  is  computer-based  training  effective  for  all  learners  and 
types  of  tasks? 

Simulation-based  training  can  promote  retention  - 
Technology  may  be  particularly  useful  in  circumstances 
where  time  resources  are  limited  and  brief  episodes  of 
skill  performance  can  provide  some  form  of  practice 
that  will  help  to  retain  competency  (Andrews  & 

Fitzgerald,  2010). 

Does  simulation-based  training  promote  skill  retention? 

If  so,  is  simulation-based  training  effective  for  all  learners 
and  types  of  tasks? 

Models  of  Skill  Retention 

A  number  of  models  have  been  developed  to  predict 
skill  retention  (Bryant  &  Angel,  2000). 

Retention  interval  -  Performance  declines  as  the 
retention  interval  is  increased  (Semb  et  al.,  1992; 

Bryant  and  Angel,  2000).  The  level  of  skill  degradation 
depends  on  the  type  of  skill 

Can  models  of  skills  retention  predict  skill  decay? 

Techniques  for  Retention 

Stronger,  more  abstract  and  developed  mental  models 
can  also  increase  retention  of  skills  (Rowatt  & 

Shlechter,  1993) 

Allowing  learners  to  develop  their  own  mnemonics 
enhances  retention  and  transfer  of  procedural  tasks 
(Druckman  &  Bjork,  1991;  Wells  &  Hagman,  1991;  cited 
in  Rowatt  &  Shlechter,  1993). 

What  retention  techniques  are  the  most  effective? 

Can  mental  models  improve  rates  of  retention? 

What  kinds  of  tools/aids  are  most  helpful  for  improving 
retention  rates?  Are  there  tools  equally  applicable  to  all 
tasks? 

Discrete  procedural  tasks  (e.g.,  job  knowledge)  are 
more  quickly  forgotten  than  more  complex  continuous 
skills  (Schaab  &  Dressel,  2001)  (e.g.,  cognitive  skills) 

Create  a  predictive  model  depicting  retention  of  varying 
types  of  skills  be  created? 
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Given  the  results  of  the  literature  review,  then,  it  was  important  to  work  with  subject  matter 
experts  in  training  and/or  learning  and  to  further  explore  the  meaning,  implications  and 
importance  of  these  topic  areas  from  their  perspectives.  Eliciting  their  views  was  critical  to 
identifying  the  most  important  topics  to  be  pursued  in  future  research  and  development  efforts 
undertaken  by  DRDC  and  CDA.  This  workshop  is  further  detailed  in  subsequent  sections. 
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3  Accelerated  Learning  and  Retention  Workshop 

3.1  Introduction 

This  section  describes  the  results  of  a  workshop  conducted  by  the  CDA  and  DRDC.  The 
obj  ectives  of  the  workshop  were  to  prioritize  the  topics  identified  in  the  literature  review,  and  to 
understand  how  stakeholders  involved  in  CF  training  view  the  requirements  of  future  research 
and  development  efforts. 

The  outcomes  will  be  used  to  formulate  an  ARP  proposal(s)  that  will  analyse,  design,  develop  and 
assess  instructional  approaches,  technologies,  strategies  and  interventions  that  have  most  promise 
for  reducing  learning  times  and  ensuring  maximal  retention  within  the  CF. 

3.2  Methods 

A  one-day  workshop  was  held  to  explore  the  15  final  topics  that  emerged  from  the  literature 
review  with  stakeholders.  This  section  details  the  stakeholders  who  participated  in  the  workshop, 
the  schedule,  and  the  method  used  to  conduct  the  workshops. 

3.2.1  Participants 

Stakeholders  who  participated  in  the  prioritization  workshop  were  selected  by  CDA  and  DRDC. 
They  were  chosen  on  the  basis  of  their  involvement,  expertise  and/or  interest  of  the  CF  training 
system.  Participants  were  from  organizations  and  agencies  including  DRDC,  CDA,  Royal 
Military  College  of  Canada  (RMCC)  as  well  as  other  Canadian  Forces  organizations  spanning  all 
three  environments  (air,  maritime,  land). 

3.2.2  Location  and  Schedule 

The  workshop  was  conducted  in  Kingston,  Ontario  on  the  campus  of  the  RMCC.  Afternoon 
activities  were  held  at  the  CDA  also  on  the  RMCC  campus.  All  materials  for  the  group  activity 
sessions  (matrices  for  rating  topics,  paper,  and  markers)  were  provided  by  CDA  facilitators. 

The  workshop  followed  a  structured  schedule,  running  from  0800  to  1600  with  a  1  hour  lunch 
break  and  two  15-minute  coffee  breaks.  The  agenda  that  was  sent  out  to  attendees  is  shown  in 
Table  10. 


Table  10  Workshop  Schedule 


Time 

Workshop 

Phase 

Event 

Speaker 

0800 

Phase  1 

Workshop  Introduction 

LCdr  Peter  Ball,  CDA 

0815 

Guest  Speaker  Introduction 

Dr.  Lochlan  Magee,  DRDC 
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Time 

Workshop 

Phase 

Event 

Speaker 

0820 

Phase  2 

Perspective  on  Accelerated  learning 

Dr.  Dee  Andrews,  USAF  Research 
Laboratory 

0835 

Phase  3 

Keynote  Address  -  Accelerated  learning 

Dr.  Robert  Hoffman,  Florida  Institute  for 
Human  and  Machine  Cognition 
(FIHMC) 

0910 

Phase  4 

Presentation  of  topics  emerging  from  literature 
review 

Dr.  Barbara  Adams,  Humansystems 

Inc. 

1025 

REFRESHMENT  BREAK 

1040 

Presentation  and  Discussion  of  Criteria 

Dr.  Lochlan  Magee,  DRDC 

LCdr  Peter  Ball,  CDA 

1100 

Phase  5 

Dry  Run 

A  dry  run  the  afternoon’s  prioritization  process  will 
be  conducted  (or  the  first  topic  conducted 
together). 

Capt.  John  Wyville,  CDA 

1200 

LUNCH  BREAK 

1300 

Phase  6 

Break-out  Group  Activity 

Each  group  will  use  the  criteria  to  score  the  topics 
and  prioritize  them  to  the  5  strongest. 

Group  Facilitators,  CDA 

1530 

Phase  7 

Workshop  Summary 

After  a  brief  break  where  the  group  facilitators  will 
consolidate  the  results  the  group  will  reconvene.  A 
presentation  will  be  given  on  the  results  of  the 
breakout  groups  and  the  overall  5  strongest  topics 
selected. 

LCdr  Peter  Ball,  CDA 

Dr.  Lochlan  Magee,  DRDC 

This  agenda  was  followed  during  the  workshop,  with  some  minor  variations  in  time.  The  seven 
phases  of  the  workshop  are  detailed  in  the  sections  that  follow. 

3.2.2. 1  Phase  1  -  Workshop  Introductions  and  Guest  Speaker  Introductions 

LCdr  Peter  Ball  provided  a  brief  introduction  of  the  purpose  and  objectives  of  the  current  work 
and  associated  workshop.  This  time  also  included  an  overview  of  the  day’s  agenda,  schedule  and 
administrative  details  (e.g.  location  of  facilities,  emergency  exits,  etc.).  The  full  introductory 
presentation  is  available  in  Annex  A-Workshop  Introductions  . 

Dr.  Lochlan  Magee  introduced  the  guest  speaker  (Dr.  Dee  Andrews)  and  keynote  speaker  (Dr. 
Robert  Hoffman);  he  provided  a  description  of  their  expertise  and  relevant  work. 
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3. 2. 2. 2  Phase  2  -  Guest  Speaker  Presentation:  Dr.  Dee  Andrews 


Dr.  Dee  Andrews,  a  senior  scientist  in  the  Human  Effectiveness  Directorate  Air  Force  Research 
Laboratory  (U.S.  AFRL)  connected  remotely  to  provide  a  presentation  on  his  work  related  to 
accelerated  learning.  His  current  work  at  the  U.S.  AFRL  demonstrates  that  accelerated  learning 
has  been  recognized  as  an  important  topic  in  improving  military  training  and  consequently 
performance.  Dr.  Andrews  has  taken  the  lead  in  conducting  this  research  working  with  a  team  of 
researchers  including  Dr.  Robert  Hoffman. 

Dr.  Andrews  argued  that  current  warfighters  are  often  required  to  perform  complex  tasks  under 
time  pressure,  even  when  they  are  not  particularly  well  trained.  Irregular  warfare  (IW), 
counterinsurgency  (COIN),  and  Stability,  Security,  Transition,  Reconstruction  Operations 
(SSTRO)  demand  “real-time  situational  understanding”.  Dynamic  planning  skills  (including  both 
kinetic  and  non-kinetic  skills)  and  interpersonal  skills  are  particularly  difficult  skills  to  learn.  Dr. 
Andrews  identified  a  number  of  accelerated  learning  challenges  that  are  not  well  funded.  These 
include  understanding  proficiency  requirements  and  how  to  achieve  accelerated  learning,  how  to 
increase  retention,  and  how  to  calculate  a  cost/benefit  analysis  when  implementing  accelerated 
learning.  Accelerated  learning  requires  constant  “stretching”  of  the  skill,  high  levels  of  intrinsic 
motivation,  enriched  practice  with  meaningful  feedback,  mentoring,  adapted  (or  tailored)  practice 
that  matches  unique  learning  styles,  the  concept  of  scheduling  (optimal  spacing  of  materials), 
testing  used  in  learning,  self-explanation,  and  learning  through  comparison.  Dr.  Andrew 
identified  a  number  of  techniques  for  accelerating  learning  emerging  from  the  cognitive  science 
literature  such  as  spacing  of  practice,  using  tests  to  promote  learning  and  use  of  deep  principles. 
Thus  far,  Dr.  Andrews  and  colleagues  have  generally  found  positive,  successful  results  from  their 
research.  He  recognized  that  specific  training  activities  such  as  “Think  like  a  Commander”  have 
demonstrated  the  ability  to  compact  years  of  military  experience  and  training  into  a  few  weeks. 
The  approach  requires  students  to  think  through  highly  complex  scenarios  reflective  of  realistic 
military  encounters.  Solving  problems  within  multiple  realistic  situations  within  a  few  weeks 
provides  experience  that  would  normally  take  years  to  accumulate.  In  addition  to  the  successes,  a 
number  of  paradoxes  exist.  First,  accelerated  learning  may  adversely  affect  retention,  so  it  will  be 
important  to  find  the  best  possible  balance  between  the  two.  Second,  accelerated  learning  in  one 
area  may  have  negative  consequences  for  generalization  to  a  different  situation.  Lastly, 
accelerated  learning  through  means  of  shorter/faster  training  may  result  in  lack  of  readiness  for 
task  performance.  Much  investigation  is  required  to  attain  accelerated  learning  within  the  military 
domain;  in  particular  research  of  longitudinal  and  long-term  retention  studies,  attention  to  policy 
issues  within  the  military,  and  funding  for  long-term  programs.  See  Annex  B-  Guest  speaker 
presentation:  Dr.  Dee  Andrews  for  the  full  presentation. 

3. 2. 2. 3  Phase  3  -  Keynote  Address:  Dr.  Robert  Hoffman 

Dr.  Robert  Hoffman  was  invited  as  the  keynote  speaker.  As  a  world  leader  in  the  field  of 
cognitive  systems  engineering  and  Human  Centered  Computing  (HCC),  Dr.  Hoffman  has  made 
significant  contributions  in  a  number  of  research  areas,  including  human  factors  in  remote 
sensing,  the  psychology  of  expertise  and  use  of  cognitive  task  analysis,  and  on  intelligent  systems 
technology  and  the  design  of  macrocognitive  work  systems.  He  is  currently  working  as  a  senior 
research  scientist  at  the  Florida  Institute  for  Human  and  Machine  Cognition  (FIHMC)  in 
Pensacola,  Florida.  Dr.  Hoffman  has  been  working  with  Dr.  Andrews  and  a  team  of  researchers  in 
investigating  accelerated  learning. 
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Dr.  Hoffman  presented  on  the  topic  of  accelerated  learning  highlighting  the  problem  space, 
challenges,  paradoxes,  potential  solutions  and  issues.  He  argues  that  the  push  to  understand  how 
to  accelerate  learning  stems  from  the  need  to  know  how  to  build  expertise  more  quickly  (in  order 
to  escape  the  “ten-year  rule”)  and  to  understand  how  to  rapidize  the  re-acquisition  of  skills  that 
have  faded.  The  need  is  to  have  experts  in  their  field  be  able  to  make  fast  decisions,  to  resolve 
tough  problems  using  good  judgment,  and  to  anticipate  future  problems.  As  Hoffman  argues,  they 
must  “...exercise  effective  technical  leadership  in  ambiguous  situations”.  However,  the 
acquisition  of  expertise  usually  takes  years.  Within  a  military  context,  factors  such  as  collateral 
assignment,  redeployment  (after  time  without  training),  inadequate  mentoring  and  “just-in-time 
training”  (rather  than  continual  training)  present  additional  challenges  to  high  proficiency.  Dr. 
Hoffman  also  noted  that  there  are  multiple  meanings  of  accelerated  learning,  including  rapidized 
training  (while  ensuring  its  effectiveness),  rapidized  knowledge  sharing  (moving  lessons  learned 
in  battlespace  into  training)  and  accelerated  proficiency.  Accelerated  learning  is  relevant  in  many 
different  domains,  including  cultural  awareness  and  understanding,  second  language  learning, 
piloting  and  personnel  management.  Dr.  Hoffman  indicated  a  number  of  success  stories,  namely 
intelligent  tutoring  systems,  Top  Gun  and  successes  in  simulation-based  training.  He  then 
indicated  a  number  of  relevant  research  results  showing  ways  to  improve  training.  Specifically,  it 
is  more  effective  when  combined  with  considerable  practice  in  scenarios  or  with  realistic 
examples,  and  when  outcome  and  process  feedback  are  provided  at  critical  stages.  Tough 
problems  and  mentoring  can  also  help  to  advance  learning.  In  terms  of  retention,  Dr.  Hoffman 
indicated  the  benefits  of  spacing  of  practice,  particularly  when  practice  on  the  target  task  is 
interspersed  with  other  tasks.  Overlearning  is  also  identified  as  a  prime  determinant  of  memory 
and  skill  decay.  However,  he  also  noted  that  the  effects  noted  in  the  literature  are  often 
contradictory,  so  generalizations  are  difficult.  A  particular  challenge  in  judging  the  effectiveness 
of  training  interventions  is  intermediate  causation,  and  the  fact  that  there  is  always  a  complex  set 
of  interacting  factors  that  impact  on  outcomes.  Dr.  Hoffman  indicated  that  some  paradoxes  are 
also  relevant  to  accelerated  learning.  Specifically,  the  paradox  of  jobs  is  that  the  nature  of  work 
constantly  changes,  so  learning  and  re-leaming  is  always  necessary.  However,  the  most  highly 
skilled  personnel  are  often  moved  to  other  positions.  The  paradox  of  tasks  is  that  they  are  often 
conceptualized  as  discrete  and  sequential,  but  they  are  actually  context-dependent. 

In  his  summary,  Dr.  Hoffman  addressed  several  questions.  He  argued  that  learning  can  be 
accelerated  for  fixed  tasks  and  probably  for  dynamic  tasks.  Research  is  currently  unclear  about 
whether  accelerated  learning  penalizes  retention  on  its  own  or  after  hiatus.  Requirements  for 
future  research  are  better  methods  for  cognitive  task  analysis,  addressing  the  challenge  of  the  pace 
of  change  in  professions  practice  (e.g.,  military  jobs),  and  better  understanding  the  potential 
contribution  of  mentors.  Other  methodological  research  needs  include  understanding  the  impact 
of  “booster  sessions”  on  retention,  differential  decay  rates,  and  ways  to  rapidly  update  skills  to 
prevent  decay.  Dr.  Hoffman  also  identified  several  acceleration  methods,  including  technologies 
such  as  computer  games  and  simulations,  case-based  instruction,  corrective  feedback  and  tough- 
case  time  compression.  Lastly,  he  identified  several  areas  that  are  the  most  important  priorities  for 
research  and  development.  These  include  longitudinal  studies,  dynamic  tasks  and  challenges, 
domains  with  civilian  analogs,  and  integrated  knowledge  management  components.  Research  that 
targets  achievement  of  proficiency  (particularly  in  the  apprentice  to  senior  journeyman  level), 
helps  to  understand  the  retention  of  expertise  at  the  level  of  personnel  as  well  as  incorporated 
organizational  knowledge,  and  with  application  to  the  military  domain  are  also  important 
priorities.  More  specific  details  of  the  presentation  can  be  found  in  Annex  C-  Keynote  address: 
Dr.  Robert  Hoffman. 
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3. 2. 2. 4  Phase  4  -  Presentation  of  Topics 

Dr.  Barbara  Adams,  part  of  the  HSIs  research  team,  presented  fifteen  research  topics  that 
emerged  from  the  literature  review  and  subsequent  conversations.  Fourteen  topics  were  identified 
in  the  review.  The  Embedded  Training  topic  was  added  after  consultation  with  the  SA  and  TA. 
Each  topic  was  defined  and  presented  with  paradigmatic  research  questions.  These  topics  were 
divided  into  learning  and  retention  categories,  as  follows: 

Learning  Topics 

1 .  Student  attributes 

2.  Instructor  attributes 

Delivery  Media 

3.  Classroom  aids/ environment 

4.  Distance  learning 

5.  Computer-based  instruction 

6.  Simulator-based  instruction 

7.  Embedded  training 

Delivery  Methods 

8.  Collaborative  learning 

9.  Scheduling 

10.  Feedback  and  testing 

1 1 .  Adaptive  instruction 

12.  Context  and  Content 

Retention  Topics 

13.  Relationships  to  learning  factors 

14.  Models  of  knowledge  /skill  retention 

15.  Techniques  for  Retention 

For  the  full  presentation  see  Annex  A. 

3. 2. 2. 5  Phase  5  -  Dry  Run 

Captain  John  Wyville  provided  an  overview  of  the  afternoon  break-out  group  activity.  The  dry 
run  involved  a  brief  discussion  and  mock  rating.  The  participants  were  informed  that  the  break¬ 
out  sessions  would  involve  a  brief  discussion  on  each  topic  in  order  to  share  their  expertise  and 
perspectives  followed  by  completion  of  a  matrix  including  ratings,  ranking  and  comments  for 
each  of  the  15  topics. 

Participants  were  provided  an  opportunity  to  briefly  discuss  the  topic  of  ‘student  attributes’  as  an 
example  discussion.  Following  the  discussion,  Captain  John  Wyville  demonstrated  how 
participants  would  complete  each  rating  using  large  print-outs  of  the  rating  matrix. 
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Rating  criteria  were  explained  as  well  as  each  of  the  1 5  evaluation  criterion.  The  rating  scale  to  be 
used  ranged  between  1  (strongly  disagree)  and  5  (strongly  agree)  with  3  (neither  agree  nor 
disagree)  as  the  midpoint.  Participants  were  asked  to  use  an  asterisk  (*)  for  ‘do  not  know’  if  they 
were  unable  to  rate  topic  areas  (e.g.,  due  to  lack  of  knowledge  or  experience).  The  15  evaluation 
criteria  were  provided  by  DRDC  and  CDA  to  reflect  the  criteria  used  to  evaluate  ARP  proposals 
within  the  DRDC  system.  The  criteria  presented  by  the  facilitator  were  as  follows: 

A.  The  topic  has  scientific  merit. 

B.  The  topic  is  novel  or  innovative  in  addressing  learning  and  retention. 

C.  The  topic  aligns  with  and  supports  CF/DND  goals. 

D.  The  topic  will  have  wide  application  within  CF/DND. 

E.  The  topic  will  improve  organizational  efficiency  and/or  effectiveness. 

F.  The  topic  will  address  identified  gaps  -  learning  and  retention  deficits  -  in  the 
CF/DND. 

G.  The  anticipated  results  of  addressing  the  topic  will  accelerate  learning  and/or 
improve  retention. 

H.  The  anticipated  results  will  be  relatively  easy  to  implement. 

I.  National  or  international  collaboration  on  this  topic  is  possible. 

J.  The  topic  will  attract  an  industrial  stakeholder  for  R&D. 

K.  There  is  sufficient  capability  and/or  capacity  within  Canada  to  take  on  this  topic 
for  R&D. 

These  criteria  were  presented  in  the  matrix  that  participants  were  required  to  complete  during  the 
afternoon  breakout  sessions.  After  ranking  each  of  these  criterion,  participants  were  also  required 
to  calculate  a  total  (sum  of  all  ratings),  and  provide  the  number  of  criteria  for  which  they  had 
provided  a  score  (i.e.,  excluding  columns  with  asterisks).  A  column  for  “rank”  was  also  included 
in  the  spreadsheet,  and  participants  were  expected  to  rank  the  importance  of  the  topics  from  their 
unique  perspective.  This  would  provide  another  way  of  comparing  the  relative  importance  of  the 
topic  areas.  Lastly,  a  column  for  participant  comments  was  also  included  in  the  matrix. 

3. 2. 2. 6  Phase  6  -  Break-out  Group  Activity 

The  afternoon  sessions  began  with  a  brief  introduction  and  recap  of  the  task.  In  all,  thirteen 
participants  were  divided  into  1  of  3  groups  by  the  primary  CDA  facilitator,  resulting  in  one 
group  of  two  participants,  a  group  of  five,  and  a  group  of  six  participants.  These  groups  were  each 
lead  by  a  CDA  facilitator,  and  discussion  centred  on  each  of  the  15  topics  identified  as  potentially 
relevant  to  learning  and  retention.  Following  the  discussion  of  each  topic,  participants  were  asked 
to  complete  the  matrix  by  rating  each  topic  area  on  each  of  the  evaluation  criteria. 

Two  participants  left  during  afternoon  session  due  to  prior  commitments,  leaving  one  group  of 
five,  a  group  of  two,  and  a  group  of  four.  All  participants  completed  ratings  prior  to  leaving  the 
workshop.  CDA  facilitators  ran  each  of  the  groups  allowing  time  for  discussion  on  each  topic  and 
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to  complete  ratings.  Participants  were  encouraged  to  write  on  their  matrices  all  comments  and 
potential  research  questions  that  came  up  during  discussion. 

One  I  IS  I  researcher  accompanied  each  break-out  group  to  collect  detailed  notes  of  the 

discussion,  and  to  provide  definitions  of  each  of  the  topics  as  necessary. 

3. 2. 2. 7  Phase  7  -  Workshop  Summary 

After  completion  of  the  break-out  sessions,  all  remaining  workshop  participants  reconvened  in  a 
common  room.  Participants’  matrices  (with  completed  topic  ratings  on  the  evaluation  criteria) 
were  collected  by  CDA  facilitators,  and  analyzed  in  a  preliminary  assessment.  As  a  conclusion  to 
the  workshop,  participants  were  brought  together  to  discuss  general  thoughts  on  the  15  research 
topics,  evaluation  criteria,  and  their  overall  experience  in  the  workshop. 

After  the  workshop,  the  data  was  collected  and  further  analyzed  by  HSI®  researchers.  Qualitative 
analyses  focused  on  bringing  together  the  group  discussions  that  had  occurred  during  the  break¬ 
out  sessions.  Quantitative  analyses  focused  on  the  ratings  and  rankings  of  each  of  the  15  topics,  as 
well  as  ratings  of  the  evaluation  criteria. 

3.3  Results 

This  section  details  the  results  of  the  accelerated  learning  workshop.  This  section  begins  with  an 
overview  of  some  of  the  limitations  of  the  workshop.  Any  scientific  activity  in  applied  settings  is 
subject  to  challenges  and  limitations.  As  some  of  the  challenges  faced  during  the  workshop 
impact  substantively  on  the  results  and  their  interpretation,  it  seems  important  to  explore  these 
before  the  rich  qualitative  analyses  and  quantitative  results. 

3.3.1  Limitations 

The  results  of  this  workshop  are  limited  in  several  ways.  As  it  was  necessary  to  postpone  and 
reschedule,  workshop  attendance  was  significantly  lower  than  initial  projections.  Due  to  prior 
commitments,  some  participants  who  attended  the  morning  session  were  not  able  to  attend  the 
afternoon.  Two  participants  also  needed  to  leave  during  the  afternoon  sessions.  Fortunately,  these 
two  participants  completed  their  ratings  prior  to  their  departure.  Although  the  workshop  provided 
rich  information,  the  small  sample  size  (n  =  13)  limits  the  analyses  that  can  be  meaningfully 
conducted  on  these  data,  and  may  limit  the  generalizability  of  the  results. 

Missing  data  also  limits  the  strength  of  the  conclusions  that  can  be  draw  from  the  workshop 
results.  Participants  were  unable  to  provide  ratings  for  many  of  the  items.  An  analysis  of  the 
missing  data  showed  that  of  the  possible  2145  data  points  (ratings  by  13  participants  on  15  topics, 
and  1 1  evaluation  criteria),  10%  of  these  data  points  did  not  receive  ratings  from  participants. 


Further  analyses  were  conducted  to  better  understand  these  missing  data  at  the  topic  level,  as 
shown  in  Table  11. 
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Table  11  Missing  data  -  topics 


# 

Missing 

ratings 

% 

Missing 

Student  attributes 

13 

9.1% 

Instructor  attributes 

10 

7.0% 

Delivery  media 

classroom  aids  and  environ 

8 

5.6% 

distance  learning 

4 

2.8% 

computer-based  instruction 

9 

6.3% 

simulator-based  instruction 

3 

2.1% 

embedded  training 

14 

9.8% 

Delivery  methods 

collaborative  learning 

16 

11.2% 

scheduling 

28 

19.6% 

feedback  and  testing 

16 

11.2% 

adaptive  instruction 

10 

7.0% 

Content  and  context 

14 

9.8% 

relation  to  learning 

48 

33.6% 

models  of  skill  retention 

11 

7.7% 

techniques  for  retention 

12 

8.4% 

As  Table  11  shows,  then,  missing  data  were  more  common  for  topics  that  participants  had 
reported  having  difficulty  understanding,  namely,  the  scheduling  and  relation  of  retention  to 
learning  factor  topics.  Further  analyses  explored  missing  data  at  the  participant  level,  and  showed 
that  missing  responses  varied  from  3%  to  22%  among  participants. 

Lastly,  missing  data  for  the  evaluation  criteria  were  also  analysed.  Missing  responses  by 
evaluation  criterion  are  shown  in  Table  12. 


Table  12  Missing  data-  evaluation  criteria 


#  Missing 
ratings 

% 

Missing 

A  -  The  topic  has  scientific  merit. 

6 

3.2% 

B  -  The  topic  is  novel  or  innovative  in  addressing  learning  and  retention. 

5 

2.6% 

C  -  The  topic  aligns  with  and  supports  CF/DND  goals. 

16 

8.5% 

D  -  The  topic  will  have  wide  application  within  CF/DND. 

9 

4.8% 

E  -  The  topic  will  improve  organizational  efficiency  and/or  effectiveness. 

14 

7.4% 

F  -  The  topic  will  address  identified  gaps  -  learning  and  retention  deficits  - 
in  the  CF/DND. 

28 

14.8% 

G  -  The  anticipated  results  of  addressing  the  topic  will  accelerate  learning 
and/or  improve  retention. 

10 

5.3% 

H  -  The  anticipated  results  will  be  relatively  easy  to  implement. 

17 

9.0% 

1  -  National  or  international  collaboration  on  this  topic  is  possible. 

24 

12.7% 

J  -  The  topic  will  attract  an  industrial  stakeholder  for  R&D. 

45 

23.8% 

K  -  There  is  sufficient  capability  and/or  capacity  within  Canada  to  take  on 
this  topic  for  R&D. 

42 

22.2% 
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This  analysis  is  again  consistent  with  participants’  comments  in  the  AAR  (Section  3.3.4)  noting 
that  some  of  the  evaluation  criteria  were  particularly  difficult  for  them  to  know  how  to  rate  (i.e.,  J, 
K).  Due  to  the  complexity  of  the  research  literature,  the  topics  were  necessarily  broad  and  often 
multidimensional.  Participants  noted  that  it  was  difficult  to  know  how  to  rate  a  topic  on  a  given 
criterion,  as  they  would  rate  the  multiple  facets  of  a  topic  differently.  Since  their  specific  mental 
models  of  the  topic  were  not  considered,  it  was  unclear  which  specific  aspect  of  a  complex  topic 
actually  influenced  their  ratings.  As  a  whole,  then,  participants  seemed  to  err  on  the  side  of 
caution  if  they  were  unsure  about  how  to  rate  the  required  elements.  Although  this  missing  data 
makes  it  difficult  to  do  extensive  analyses  on  this  limited  data  set,  the  ratings  that  they  did 
provide  hopefully  reflect  their  clear  areas  of  experience  and  expertise  with  low  levels  of 
speculation. 

The  lack  of  consistency  within  the  breakout  group  discussion  is  a  potential  limitation.  Participants 
were  provided  with  general  instructions  as  a  large  group;  however,  there  was  little  consistency  in 
terms  of  how  the  break-out  groups  were  conducted.  Hence,  participants  might  have  had  different 
experiences.  For  example,  ensuring  that  participants  understood  exactly  how  the  terms  were  being 
defined  occurred  differently  within  the  3  groups,  and  this  could  have  impacted  how  ratings  were 
completed.  As  discussed  in  Section  3.3.2  below,  for  example,  one  group  was  left  to  discuss  topics 
as  they  naturally  evolved,  rather  than  the  other  two  groups  where  the  facilitator  actively  promoted 
discussion  and  progressed  through  topics  systematically. 

It  might  also  have  been  helpful  to  have  some  background  information  about  participants,  and  to 
understand  their  particular  stake  and  expertise  in  the  workshop  topics.  Given  the  apparent 
differences  in  facilitation  style  within  the  break-out  groups,  it  might  have  been  helpful  to  evaluate 
how  ratings  related  to  group  membership  and  to  the  experiences  within  the  group.  However, 
which  questionnaires  belonged  to  which  group  members  was  unknown.  Lastly,  having  all  groups 
work  through  the  topics  in  the  same  order  might  have  resulted  in  the  early  topics  getting 
considerably  more  discussion  than  later  topics,  due  to  either  time  constraints  or  participant 
fatigue.  The  amount  of  discussion  with  later  topics  seemed  to  have  diminished  somewhat  as 
groups  worked  through  the  topics. 

Of  course,  it  is  important  to  note  that  any  applied  research  effort  is  likely  to  face  challenges,  and 
that  there  is  no  way  to  know  the  extent  to  which  these  limitations  impacted  the  results  (if  at  all). 
Comments  from  participants  in  the  after  action  review  were  strongly  supportive  of  this  workshop, 
and  it  did  elicit  important  discussion  and  debate  about  the  most  critical  factors  related  to  learning 
and  retention.  As  such,  even  with  the  potential  limitations,  this  research  provided  a  good 
empirical  base  for  advancing  these  topics  in  future  research  and  development  efforts. 

3.3.2  Qualitative  Results 

Some  general  observations  seem  important  to  understanding  the  rich  data  that  emerged  from  the 
breakout  group  discussions.  At  the  start,  the  breakout  groups  received  somewhat  different 
instructions.  One  group  was  instructed  that  the  15  topics  should  be  considered  in  the  light  of 
relevant  implementation  within  the  CF,  as  well  as  to  include  multigenerational  learners  (e.g.,  17- 
55  years  old),  different  experiences  and  different  learning  styles.  The  second  group  was  provided 
encouragement  to  discuss  each  topic  as  it  best  related  to  CF  training  in  general.  Participants  in  the 
third  and  final  group  were  provided  little  instruction  other  than  being  encouraged  to  talk  about 
topics  as  they  saw  fit.  A  novel  approach  taken  by  one  of  the  facilitators  was  to  combine  some  of 
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the  topic  areas  during  the  discussion  (e.g.,  simultaneous  discussion  of  student  attributes  and 
instructor  attributes  and  talking  about  the  issue  of  retention  as  a  whole). 

Within  2  groups,  facilitators  guided  participants  to  move  from  topic  to  topic,  whereas  discussion 
in  the  3rd  group  was  guided  by  the  two  participants.  As  groups  shifted  to  new  topics  within  the  2 
groups,  Humansystems®  observers  recorded  the  amount  of  time  spent  on  each  topic  within  the 
available  2  groups.  Discussions  on  topics  ranged  from  0  (i.e.,  some  topics  received  no  discussion 
due  to  time  constraints)  to  1 5  minutes,  with  an  average  of  6  minutes  on  each  topic. 

During  discussions,  groups  often  requested  definitions  of  the  topics,  and  these  were  provided  by 
Humansystems ®  observers  based  on  the  PowerPoint  presented  by  Humansystems 8  in  the  morning 
session  (  Annex  D-  Presentation  of  topics). 

3. 3. 2.1  Student  attributes 

As  noted  in  the  literature  review,  prominent  dimensions  of  student  attributes  encompass  cognitive 
ability,  motivation,  learning  styles,  and  prior  knowledge  or  experience  (e.g.,  training  and 
education). 

As  participants  discussed  student  attributes,  they  noted  that  this  was  a  broad  topic  that 
encapsulated  social  phenomenon  (i.e.,  culture,  generational  differences),  technology,  individual 
characteristics  (i.e.,  motivation,  individual  variation,  ADD),  and  program  development  (i.e.,  target 
audience). 

First,  participants  discussed  how  social  phenomenon  such  as  culture  could  be  leveraged  to 
accelerate  certain  types  of  training.  Also,  participants  noted  that  with  an  increase  in  technology, 
the  style,  and  pace  of  learning  have  dramatically  increased  among  younger  generations.  Multi¬ 
tasking  may,  in  some  cases,  be  required  in  order  to  keep  the  attention  of  today’s  youth. 

Further,  participants  noted  that  training  specific  technology  has  significantly  developed  over  the 
past  20  years.  Familiarity  with  social  media  and  comfort  with  technology  differs  among 
generations.  How  technology  should  be  implemented  would  naturally  need  to  vary  depending  on 
the  target  generation,  and  they  noted  that  this  issue  warrants  further  research.  They  further  noted 
that  although  there  is  some  research  available  on  these  topics,  very  little  is  related  to  military 
training  in  general,  and  specifically  to  the  CF. 

In  addition  to  social  phenomenon,  individual  characteristics  were  discussed.  Motivation  was 
recognized  as  being  vital  to  accelerate  learning  and  increase  retention  by  two  of  the  three  groups. 
Participants  were  interested  in  discussing  potential  factors  that  influence  high  motivation.  It  was 
suggested  that  factors  within  the  training  program  or  environment  (e.g.,  instructor  attributes,  or 
technology)  may  influence  trainees’  motivation.  One  method  to  increase  trainee  motivation  could 
be  to  focus  on  career  planning  as  motivation.  For  example,  allowing  trainees  to  have  more 
influence  over  their  pay  grades,  promotions,  and/or  postings  based  on  success  or  achievement  in 
training  programs.  Participants  voiced  concern  that  implementation  of  such  programs  may  be 
significantly  affected  by  the  systemic  or  organizational  limitations  (and  vice-versa). 

The  role  of  student  attributes  in  training  preparations  was  also  discussed.  Notably,  in  the  design 
phase  of  creating  training  programs,  some  participants  noted  that  target  audiences  are  not  often 
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taken  into  account.  The  variation  among  trainees  may  be  too  large  to  consider  when  developing  a 
course.  Questions  were  raised  as  to  how  one  would  make  a  training  program  more  flexible  in 
order  to  accommodate  a  wide  range  of  student  attributes.  The  current  system  does  not  take 
trainees’  previous  experiences  or  special  needs  (e.g.,  attention  deficit  disorder)  into  account. 
Trainees  who  start  with  transferable  skills  may  be  able  to  move  through  training  at  a  quicker  pace, 
or  start  further  along  in  the  training.  Individuals  who  have  special  needs  may  require  additional 
effort  and  time  in  certain  areas  in  order  to  achieve  proficiency  and  retain  knowledge  and 
information.  Developing  specific  training  regimes  and  programs  is  expected  to  require 
significantly  more  effort,  which  would  in  turn  require  additional  resources  on  the  part  of  the 
developers,  instructors,  and  /or  trainees.  Participants  noted  that  it  would  be  important  to  consider 
CF  training  as  adult  learning  in  a  system  where  personnel  have  additional  responsibilities  and 
commitments  outside  a  training  program  (e.g.,  duties,  families). 

A  final  comment  noted  by  participants  was  that  the  current  programs  may  train  in  particular  ways 
that  have  not  been  assessed  or  validated.  For  instance,  CF  often  provides  background  information 
to  trainees  prior  to  the  start  of  a  training  course.  A  pre-course  reading  package  is  intended  to  bring 
trainees  up  to  a  particular  level  prior  to  the  start  of  the  course.  Flowever,  there  is  currently  no 
empirical  evidence  that  confirms  that  this  information  is  actually  helpful. 

Research  questions  relevant  to  student  attributes  during  the  break-out  discussions  are  shown  in 
Table  13. 


Table  13  Student  attributes-  Research  questions 


Research  Topic 

Research  Questions 

Student  attributes 

Can  cultural  differences  be  utilized  to  accelerate  certain  types  of  training? 

•  Does  multitasking  improve  overall  productivity  for  the  new 
generation? 

o  If  informal  learning  takes  place  in  multitasking 
environments,  what  are  the  effects  on  retention? 

•  What  causes  high  motivation?  Is  it  the  student’s  intrinsic  motivation, 
influence  of  the  instructor,  or  classroom  aids? 

•  How  does  accounting  for  student  attributes  in  practice  schedules 
influence  training  effectiveness? 

3. 3. 2. 2  Instructor  attributes 

Instructors  have  a  key  role  to  play  in  training.  Instructors  are  obviously  very  diverse,  and  factors 
such  as  their  subject  matter  expertise,  their  experience  as  a  teacher  and  their  style  of  teaching 
emerged  from  the  literature  review  as  important  instructor  attributes. 


Group  discussions  about  instructor  attributes  focused  on  the  ability  to  teach  effectively,  teaching 
style  and  preferences,  and  operational  currency.  Participants  noted  that  effective  learning  may  be 
dependent  on  the  instructor’s  ability  to  effectively  engage  and  train  students.  Future  research 
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could  focus  on  identifying  attributes  that  make  an  effective  instructor  that  could  then  be  used  for 
instructor  selection  and  training  within  the  CF.  CF  members  are  often  posted  as  instructors,  and 
although  they  may  be  experts  in  their  field,  they  are  not  necessarily  good  mentors.  Participants 
argued  that  developing  positive  instructor  attributes,  by  focusing  on  training  instructors  and  by 
providing  them  with  optimal  support,  resources,  and  networks  may  result  in  accelerated  learning 
and  benefit  retention. 

Timing  of  instructor  training  and  application  of  training  was  also  raised  as  an  issue.  Currently, 
instructor  training  is  provided,  however  the  lag  between  receiving  training  and  actually  teaching 
can  be  weeks,  months,  or  even  years.  What  impact  (if  any)  this  has  on  instructor  ability  is  not  well 
understood. 

Participants  also  noted  that  instructor  preferences  may  also  influence  the  effectiveness  of  a 
training  program.  Some  instructors  may  be  better  in  a  face-to-face  classroom  while  others  teach 
most  effectively  through  online  programs.  A  research  possibility  noted  by  participants  involves 
investigating  the  context  of  instruction  (class,  distance,  mentoring)  with  respect  to  the  instructor’s 
capabilities  and  preferences  and  the  consequent  effect  of  these  issues  on  accelerated  learning  and 
retention. 

Finally,  CF  instructors  are  posted  to  teaching  positions,  during  which  time  they  may  not  be  kept 
current  with  operational  activities  (e.g.,  new  /revised  TTPs).  Keeping  instructors  current  in 
operational  activities  may  assist  in  getting  essential  up-to-date  information  out  to  trainees.  In 
addition,  these  efforts  may  assist  in  decreasing  the  time  it  takes  to  distribute  essential  information 
to  troops  being  deployed.  Research  questions  related  to  instructors  are  noted  in  Table  14. 


Table  14  Instructor  attributes  -  Research  questions 


Research  Topic 

Research  Questions 

Instructor  attributes 

•  What  are  effective  instructor  attributes  and  how  can  they  be  trained? 

•  How  might  the  selection  of  instructors  be  influenced  based  on 
effective  instructor  attributes? 

•  Can  the  context  or  of  instruction  (class,  distance,  mentoring)  be 
matched  to  instructor  preferences  and  capabilities  in  order  to 
accelerate  training? 

•  How  does  operational  currency  (or  lack  thereof)  influence  the  ability 
to  effectively  train  students?  Could  improving  operational  currency 
be  incorporated  into  a  training  program  in  a  fashion  that  would 
accelerate  learning  and/or  improve  retention? 

3. 3. 2. 3  Delivery  Media 

Delivery  media  utilized  in  training  has  advanced  with  the  technological  boom.  As  was  noted  in  all 
groups,  the  distinction  between  classroom  aids,  computer-based  instruction,  simulation-based 
instruction,  and  embedded  training  is  ambiguous  as  these  terms  may  all  relate  to  similar 
technologies.  Distance  learning  may  also  capitalize  on  such  technologies.  Thus,  it  may  be 
beneficial  to  understand  the  ideas  that  came  from  discussion  of  these  topics  as  a  whole.  Research 
questions  that  were  identified  from  one  topic  may  be  relevant  to  another. 
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As  a  general  comment,  participants  voiced  concern  that  cost  and  technical  knowledge  may  be  an 
obstacle  to  implementation  of  different  delivery  media. 


3. 3. 2. 3.1  Classroom  Aids  and  Environment 

Classroom  aids  and  environment  describe  tools  and  different  aids  (e.g.,  concept  maps,  videos) 
that  can  be  used  within,  and  outside  of,  a  classroom  environment.  The  environment  considers 
physical  (e.g.,  noise,  lighting)  and  psychological  (e.g.,  learning  culture)  aspects  of  the  learning 
environment  that  impact  on  trainees’  ability  to  learn. 

Participants  noted  that  the  use  of  classroom  aids  seems  to  be  much  more  prominent  in  civilian 
society  than  within  the  CF.  Two  of  the  three  groups  noted  that  classroom  aids  have  not  been 
easily  implemented  into  CF  training  for  various  reasons.  The  first  is  that  there  is  a  lack  of  strong 
evidence  that  classroom  aids  may  be  useful.  Second,  the  CF  may  not  have  the  resources  necessary 
to  acquire  the  aids.  Lastly,  there  is  a  lack  of  knowledge  and  or  resources  to  optimally  implement 
classroom  aids.  Participants  noted  that  there  may  be  cases  where  the  CF  has  the  most  updated 
classroom  aids;  however,  they  have  not  been  evaluated  to  assess  how  they  can  best  be  used  to 
achieve  accelerated  learning  and  increased  retention. 

One  concern  noted  references  the  instructor’s  role.  Classroom  aids  may  require  more  effort  from 
the  instructor  to  be  successful.  This  may  not  be  feasible  given  that  instructors  often  have 
additional  responsibilities,  tasks,  and  family  life.  The  effective  implementation  of  a  classroom  aid 
may  also  be  highly  dependent  on  the  instructor’s  knowledge  and  teaching  style.  Consequently, 
using  aids  may  mean  that  instructors  may  also  require  more  training,  which  may  or  may  not  be 
feasible. 

With  respect  to  the  classroom  environment,  one  participant  was  particularly  interested  in  the 
simple  aspects  of  the  learning  environment  such  as  ergonomics  of  the  learning  space  (desk,  chair 
orientation  and  comfort),  field  of  view  within  the  classroom,  acoustics,  and  lighting.  Fie  pointed 
out  that  there  are  limited  studies  on  the  sensory  aspects  of  the  classroom.  But  this  participant 
noted  that  these  issues  could  significantly  impact  how  trainees  are  able  to  interact  with  each  other, 
with  the  instructor,  and  available  equipment.  Further  research  in  this  area,  he  argued,  may  result 
in  accelerated  learning.  Questions  that  emerged  from  the  discussion  and  participant  comments  are 
noted  in  Table  15. 


Table  15  Classroom  aids  and  environment  -  Research  questions 


Research  Topic 

Research  Questions 

Classroom  aids  / 
Environment 

What  aids  currently  exist  in  civil  society  that  effectively  accelerate  learning? 

Can  these  aids  be  feasibly  implemented  within  the  current  CF  system  and 
resources? 

What  makes  classroom  aids  effective  for  accelerated  learning? 

Is  it  the  ability  of  the  instructor  to  effectively  implement  it? 

Is  it  motivation  of  the  student  that  is  increased  by  its  use? 

Does  the  CF  have  classroom  aids?  If  so,  can  they  be  evaluated,  or 
implemented  differently  in  order  to  achieve  accelerated  learning  and  improve 
retention. 
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3. 3. 2. 3. 2  Distance  learning 

Distance  learning  can  be  defined  as  learning  that  occurs  outside  of  the  traditional  co-located 
classroom  involving  a  separation  of  distance  and/or  time  between  the  source  of  the  information 
and  learners. 

Benefits  and  cautions  related  to  distance  learning  were  identified  throughout  discussions.  Overall 
training  time  may  be  reduced  with  the  implementation  of  distance  learning  due  to  reduced  travel 
time  and/or  increased  instructor  time.  Specific  concerns  about  distance  learning  included  student 
focus,  performance  standards,  and  the  currency  of  distance  learning  research.  These  are  each 
discussed  in  turn  below. 

Distance  learning  can  reduce  the  time  allocated  to  a  course  simply  by  eliminating  travel  time 
(e.g.,  one  participant  mentioned  a  course  in  Gagetown  that  was  completed  in  3  days  through 
distance  learning  rather  than  the  typical  5  day  face-to-face  course  that  includes  travel). 

Teasing  apart  whether  certain  distance  learning  sessions  led  to  accelerated  learning  due  to 
“distance  learning”  factors  or  “better  instructional  design”  factors  was  also  discussed.  Some 
distance  learning  instructors  have  been  noted  to  spend  significantly  more  time  creating  their 
lessons  while  others  do  not.  If  no  extra  time  is  allotted,  distance  learning  can  become 
‘information  dumping’  that  may  not  be  effective.  One  group  was  specifically  interested  in 
separating  out  these  success  factors. 

A  caution  noted  about  distance  learning  is  that  students  are  often  expected  to  do  distance  learning 
in  addition  to  regular  activities.  This  takes  away  time  and  focus  on  that  material  that  would  be 
present  in  classroom  sessions.  Exactly  how  distance  learning  is  implemented,  then,  may  be  vital. 

An  additional  concern  about  distance  learning  expressed  by  participants  was  that  performance 
standards  can  easily  slip  if  evaluations  are  not  effectively  implemented.  All  aspects  of  the  testing 
situation  need  to  be  considered  in  order  to  mitigate  against  learning  violations  (e.g.,  it  may  be 
easier  for  trainees  to  cheat)  and  ensure  that  material  is  effectively  evaluated  (e.g.,  simple  multiple 
choice  tests  may  not  effectively  test  learned  material). 

Considering  current  research,  a  number  of  participants  were  concerned  that  studies  and 
knowledge  of  distance  learning  may  be  significantly  out-dated.  Improvements  in  technology  have 
changed  the  way  generations  acquire  knowledge,  and  the  speed  at  which  this  can  be  done. 
However,  they  argued  that  tools  to  best  facilitate  distance  learning  should  be  investigated. 
Questions  that  emerged  from  the  discussion  and  participant  comments  are  noted  in  Table  16. 


Table  16  Distance  learning  -  Research  questions 


Research  Topic 

Research  Questions 

Distance  Learning 

How  does  one  design  a  distance  learning  course  with  a  superior  instructional 
design  that  can  accelerate  learning? 

If  courses  are  developed  with  a  superior  instructional  design,  can  they  be 
successful  at  accelerating  learning  while  having  the  additional  benefit  of 
eliminating  travel  time? 

What  is  the  best  implementation  of  distance  learning? 
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Research  Topic 

Research  Questions 

What  are  the  best  tests  for  distance  learning? 

How  are  tests  to  validate  distance  learning  best  implemented? 

Are  there  better  ways  to  implement  distance  learning  given  new  tools  and 
technology? 

3. 3. 2. 3. 3  Computer-based  instruction 

Computer-based  instruction  is  increasingly  used  for  delivering  training.  It  has  varying  levels  of 
complexity,  ranging  from  low  (e.g.,  typing  tutor)  to  intelligent  tutoring  systems. 

In  each  group,  participants  agreed  that  computer-based  instruction  overlapped  with  classroom 
aids,  simulation-based  learning,  and  distance  learning.  Participants  were  unclear  about  the 
relationships  among  many  different  forms  of  learning,  for  example,  whether  computer-based 
instruction  encapsulated  computer-assisted  instruction  (e.g.,  students  at  their  own  computer  in  a 
classroom  with  an  instructor,  intelligent  tutors)  and  how  it  related  to  gaming  environments.  Both 
computer-based  instruction  software  and  gaming  were  discussed. 

Participants  recognized  that  computer-based  instructional  software  increasingly  involves  not  just 
one  person  and  one  program  but  multiple  people  and  virtual  spaces  (e.g.,  web-based  discussion 
board).  Such  asynchronous,  distributed  collaboration  is  a  large  benefit  of  using  computer-based 
instruction.  In  addition,  participants  noted  that  computer-based  instruction  can  be  self-paced, 
however  we  do  not  know  if  this  is  beneficial  or  detrimental  to  learning  and  retention  when  this 
form  of  instruction  would  be  maximally  beneficial.  There  is  similar  concern  with  the  timing  of 
when  asynchronous,  distributed  gaming  simulations  would  be  effectively  implemented. 

One  participant  suggested  that  research  has  not  shown  significant  differences  between  face-to- 
face  and  computer-based  learning,  thus  the  benefit  of  implementing  such  systems  should  be 
investigated  in  the  particular  training  program  context. 

A  benefit  of  gaming  environments  was  noted.  Specifically,  gaming  environments  can  provide 
experiential  learning  that  is  motivating,  and  focuses  on  soft  skills  such  as  leadership  (e.g.,  World 
of  Warcraft  requires  good  leadership  and  team  effort  in  order  to  defeat  the  enemy).  Questions  that 
emerged  from  the  discussion  and  participant  comments  are  noted  in  Table  17. 


Table  1 7  Computer-based  instruction  -  Research  questions 


Research  Topic 

Research  Questions 

Computer-based 

instruction 

What  computer-based  instruction  tools  exist  that  may  be  beneficial  to  the  CF? 

In  which  cases  is  self-paced  computer-based  instruction  beneficial? 

In  which  cases  is  computer-based  instruction  that  is  systematically  paced 
beneficial? 

What  skills  can  most  effectively  be  trained  through  computer-based  instruction? 

Is  there  current  research  that  demonstrates  the  benefit  of  computer-based 
instruction? 
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3. 3. 2. 3. 4  Simulation-based  instruction 


Simulator-based  instruction  is  an  increasingly  prominent  method  of  training  delivery.  The 
literature  defines  a  simulation  as  a  working  representation  of  reality  that  shows  devices  and 
processes,  and  provides  cues  to  elicit  responses  from  the  learner.  A  common  distinction  is 
between  live  simulation  (real  people  using  real  systems),  virtual  simulation  (real  people  in 
simulated  systems)  and  constructive  simulation  (typically  simulated  people  and  simulated 
systems). 

Discussions  on  simulation-based  instruction  touched  on  a  number  of  different  ideas;  namely, 
benefits  and  concerns  of  simulation  implementation,  topics  for  which  simulations  could  be 
applied,  blurring  of  virtual  and  reality,  acquisition  of  simulation  equipment,  and  simulation-based 
curriculum  development. 

Participants  agreed  that  the  benefits  of  using  simulations  would  typically  outweigh  any  potential 
negatives.  Simulations  may  be  used  in  team  tasks,  team  collaboration  training  as  well  as 
individual  tasks.  Participants  noted  that  the  infantry  have  used  simulations  which  allow  training  to 
proceed  as  scheduled  with  no  need  to  consider  weather.  Different  terrains  can  be  easily  modeled 
to  reflect  the  area  of  operations,  and  training  can  be  repeated,  recorded,  and  analyzed. 

Participants  also  explored  some  concerns  about  the  realism  of  simulation  (e.g.,  removing  the  need 
to  consider  weather).  The  option  to  train  under  harsh  conditions  is  not  available  in  many  of  the 
currently  used  simulations.  Similarly,  there  are  both  benefits  and  concerns  about  the  potential  for 
reducing  training  time  through  simulations.  Physical  and  psychological  fatigue  that  accumulates 
over  long  periods  of  real-time  training  would  not  be  experienced  with  simulator  training.  Some  of 
these  aspects  of  realism  that  cannot  be  provided  within  simulators  may  be  required  for  effective 
training. 

Participants  also  noted  that  virtual  and  simulated  space  is  increasingly  becoming  a  part  of  real, 
daily  life.  The  virtual  world  of  on-line  technology  is  increasingly  becoming  blurred  with  the  real 
world  through  the  use  of  distributed  on-line  tools  such  as  Twitter  and  Facebook. 

Additional  concerns  were  raised  that  the  up-front  costs  associated  with  simulation  implementation 
may  be  high  (e.g.,  financial  costs,  facilities  and  personnel  resources).  Simulations  seem  to  be 
acquired  and  used  only  for  a  focused  purpose.  In  future,  it  seems  more  effective  to  acquire 
simulation  technology  flexible  enough  to  assist  in  a  variety  of  training. 


Specific  topics  that  could  be  used  leverage  simulation-based  instruction  in  the  CF  include 
language  training  and  team  leadership  (i.e.,  learning  not  to  micro-manage),  particularly  in  the 
infantry.  Participants  argued  that  simulation-based  instruction  should  emphasise  soft  skills  (this 
was  also  mentioned  in  relation  to  computer-based  instruction)  rather  than  only  the  accumulation 
of  technical  skills  or  hard  skills. 

Finally,  participants  noted  that  simulation  developers  are  not  always  knowledgeable  about  the 
types  of  learning,  and  learning  objectives  that  takes  place  in  each  training  program.  Thus, 
developers  may  not  realize  the  full  potential  of  simulation  implementation.  Participants  also  noted 
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that  perhaps  contributions  from  trainers  and  trainees  during  the  development  of  simulations  and 
courses  would  result  in  more  effective  instruction. 


Research  questions  that  emerged  from  the  discussion  and  participant  comments  are  noted  in  Table 
18. 


Table  18  Simulator-based  instruction  -  Research  questions 


Research  Topic 

Research  Questions 

Simulation-based 

instruction 

What  type  of  simulation  is  best  for  what  context? 

What  aspects  of  realism  are  lost  in  using  simulation-based  instruction?  Are 
these  essential  in  training?  If  so,  how  can  their  loss  be  mitigated? 

Can  simulations  be  used  to  effectively  train  soft  skills? 

How  does  virtual  online  'realities’  of  simulations  effect  training,  performance, 
learning,  and  retention? 

Can  simulation  technology  be  more  flexible  /  versatile  in  a  variety  of  training? 

Can  training  curricula  be  more  effectively  developed  with  designers  and 
trainers  are  involved  in  multiple  stages  of  the  creation  and  implementation  of 
simulation  based  instruction? 

3. 3. 2. 3. 5  Embedded  training 

Embedded  Training  is  defined  as  training  that  is  built  into  or  added  to  a  weapon  system  (Witmer 
&  Knerr,  1996).  Another  commonly  used  definition  is  provided  by  the  U.S.  Army  Training  and 
Doctrine  Command,  which  defines  embedded  training  as  “A  function  hosted  in  hardware  and/or 
software,  integrated  into  the  overall  equipment  configuration.  Embedded  Training  supports 
training,  assessment,  and  control  of  exercises  on  operational  equipment,  with  auxiliary  equipment 
and  data  sources,  as  necessary”  (TRADOC  Pamphlet  350-37). 

As  a  general  observation,  participants  commented  on  their  lack  of  knowledge  and  experience  with 
embedded  training.  With  additional  definition  (based  on  the  PowerPoint  presentation)  provided 
by  HS1®  researchers,  participants  seemed  to  agree  there  is  potential  for  a  lot  of  benefit  to  emerge 
from  embedded  training.  The  ability  to  switch  into  training  mode  during  lulls  in  a  mission  (i.e. 
when  there  is  relatively  little  action)  would  increase  efficiency  and  perhaps  increase  performance 
and  reduce  additional  re-training  time.  This  type  of  switching  between  modes  is  seen  in  video 
games  such  as  X-box  and  Wii.  Embedded  training  may  also  make  it  easier  to  collect  and  utilize 
real  operational  data  for  training. 

Though  participants  admittedly  did  not  have  extensive  knowledge  about  what  is  involved  in 
embedded  training,  it  was  suggested  that  embedded  training  seems  only  relevant  to  complex  tasks 
rather  than  simple  tasks. 

Some  concern  was  raised  that  there  may  be  increased  risk  and  cost  associated  with  using 
equipment  in  training.  Research  questions  that  emerged  from  the  discussion  and  participant 
comments  are  noted  in  Table  19. 
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Table  19  Embedded  training  -  Research  questions 


Research  Topic 

Research  Questions 

Embedded  Training 

How  can  embedded  training  be  utilized  within  the  CF? 

How  can  embedded  training  be  implemented  effectively  in  the  CF  to  achieve 
accelerated  learning  and/or  increase  retention? 

3. 3.2.4  Delivery  Methods 

Delivery  methods  discussed  include  a  number  of  training  methodologies  and  techniques  that 
include  collaborative  learning,  scheduling,  feedback  and  testing,  adaptive  instruction,  and  context 
and  content. 

3. 3. 2. 4.1  Collaborative  learning 

Participants  discussed  the  nature  of  collaboration  in  a  team  or  group  within  the  CF,  information 
flow,  and  social  aspects  of  collaboration  that  may  affect  learning.  The  use  of  technology  to 
promote  collaborative  learning  was  discussed. 

Teams  within  the  CF  are  currently  taught  to  perform  as  a  team  rather  than  learn  as  a  team. 
Participants  suggested  that  team  training  should  target  how  to  enhance  the  process  of 
collaborative  learning.  Output  from  such  training  may  result  in  trainees  learning  more  when  they 
conduct  any  team  task  (in  training  or  performance). 

Participants  noted  that  collaborative  learning  provides  opportunities  that  require  trainees  to 
participate  in  order  to  share  expertise.  As  a  benefit,  all  participants  share  in  each  other’s  expertise. 
On  the  other  hand,  as  team  training  is  currently  implemented,  even  if  collaboration  is  a  focus,  not 
all  trainees  get  similar  training  experiences.  For  example,  when  demonstrations  are  used  in 
training,  trainees  play  certain  roles  and  are  only  able  to  experience  the  task  from  that  perspective. 
Timing  and  resources  do  not  allow  trainees  to  act  in  every  role. 

Collaborative  learning  may  be  affected  if  information  flow  within  the  system,  or  among  team 
members,  is  hindered.  Participants  in  one  group  were  concerned  that  the  general  culture  about 
information  sharing  is  not  always  conducive  to  collaboration.  The  military,  however,  strives  for  a 
family-type  culture  in  which  sharing  information  between  members  is  regarded  as  an  important 
feature.  It  may  be  important  to  ensure  that  information  sharing  between  echelons  and  branches 
(e.g.,  air,  maritime,  land)  of  the  CF  is  facilitated. 

Participants  noted  that  there  are  several  areas  in  which  we  know  relatively  little  about 
collaborative  learning.  Notable  gaps  include  the  use  of  social  networking  and  social  media  as  they 
relate  to  CF  training.  The  social  aspect  of  collaborative  learning  as  well  as  past  training 
experiences  (e.g.,  trained  individually  or  as  a  group)  are  anticipated  to  promote  learning.  Research 
questions  that  emerged  from  the  discussion  and  participant  comments  are  noted  in  Table  20. 
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Table  20  Collaborative  learning  -  Research  questions 


Research  Topic 

Research  Questions 

Collaborative  Learning 

Should  military  teams  be  taught  to  learn  together  as  a  team  rather  than 
simply  work  together  as  a  team?  If  so,  how  is  this  best  accomplished? 

How  can  collaborative  and  team  learning  be  implemented  to  maximize 
training  experiences  for  all  trainees  within  the  limitations  of  resources  (e.g., 
time,  equipment)? 

Does  information  flow  effectively  throughout  the  CF  in  order  to  optimize 
training?  How  can  this  be  improved?  How  can  this  be  insured? 

What  are  the  social  aspects  of  collaborative  learning  that  may  affect 
learning?  How  is  it  best  to  consider  them  to  optimize  learning  and  retention? 
What  type  of  technology  (e.g.,  social  networking,  social  media)  plays  a  role  in 
collaborative  learning?  How  might  it  be  leveraged  for  training? 

3. 3. 2. 4.2  Scheduling 

Understanding  how  to  schedule  learning  and  practice  may  be  an  important  way  to  promote 
accelerated  learning  and  retention. 

Participants  discussed  different  perspectives  of  scheduling  that  may  or  may  not  be  feasible.  They 
also  noted  scheduling  inefficiencies  within  the  current  system  that  affect  learning  and  retention. 
Informal  and  formal  training  were  discussed,  as  was  the  influence  of  technology. 

Participants  noted  the  difference  between  sequencing  courses  and  scheduling  a  course. 
Sequencing  relates  to  the  order  of  courses,  or  of  the  material  within  a  course,  whereas  scheduling 
a  course  relates  to  the  timing  of  learning.  Participants  suggested  that  optimizing  sequencing 
courses  may  improve  retention.  One  way  to  do  this  is  through  a  meta-examination  of  how  best  to 
sequence  courses  in  order  to  optimize  learning  and  retention.  However,  participants  also 
expressed  concerns  that  the  sequencing  of  courses  may  require  systematic  changes  within  the 
training  system  that  would  be  difficult  to  achieve. 

Current  scheduling  inefficiencies  within  the  system  may  negatively  affect  performance.  For 
instance,  current  training  is  often  provided  at  a  particular  time  and  not  used  until  months  or  years 
later.  The  delay  between  learning  and  application  may  severely  affect  retention  of  material  and 
consequently  performance.  Scheduling  training  appropriately  so  it  is  timely  for  application  while 
reducing  the  skill  retention  interval,  may  benefit  performance  and  help  to  reduce  skill  fading. 

Participants  also  discussed  current  issues  within  the  CF  concerning  scheduling  issues  around 
formal  and  informal  mentoring.  Mentoring  programs  and  informal  learning  may  be  beneficial; 
however,  formally  scheduling  these  types  of  learning  may  detract  from  some  of  potential  benefits. 
The  nature  of  informal  learning  is  not  conducive  to  a  natural  progression  of  knowledge  transfer. 

Participants  agreed  that  scheduling  is  an  important  topic  that  could  be  researched  further  to 
benefit  accelerated  learning  and  retention.  With  so  many  new  methods  of  learning  (e.g.,  distance 
learning,  embedded  training,  simulation  and  computer  based  learning),  there  are  many  more 
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scheduling  options  that  have  likely  not  been  thoroughly  researched.  Research  questions  that 
emerged  from  the  discussion  and  participant  comments  are  noted  in  Table  21. 


Table  21  Scheduling  -  Research  questions 


Research  Topic 

Research  Questions 

Scheduling 

How  does  scheduling  effect  learning? 

How  can  the  sequencing  of  training  programs  be  optimized  to  lead  to 
accelerated  learning  and/or  improved  retention? 

Are  there  inefficiencies  within  the  current  system  that  negatively  impact  on 
learning,  retention  and  consequently  performance? 

Can  scheduling  be  improved  to  reduce  the  delay  between  training  and 
application  of  skills? 

How  can  scheduling  be  optimized  to  accelerate  learning  given  the  extensive 
number  of  opportunities  presented  through  technology? 

3. 3. 2. 4.3  Feedback  and  testing 

Feedback  can  come  from  any  number  of  sources  (e.g.,  peers,  instructors,  computer  systems)  and 
with  varying  timing  and  frequency. 

Participants  discussed  ideas  about  the  use  and  type  of  feedback  and  testing,  associated  instructor 
responsibilities,  and  possible  techniques.  Participants  argued  that  the  idea  that  feedback  and 
testing  may  be  used  as  a  source  of  learning  and  as  a  part  of  training  may  be  an  opportunity  for 
further  research.  Possible  questions  include  how  such  testing  may  affect  retention,  and  how 
timing  of  testing  and  feedback  may  affect  learning  and  retention.  For  instance,  participants 
discussed  their  observations  that,  in  some  courses,  an  ‘attend  only’  requirement  may  result  in 
trainees  taking  the  material  less  seriously  and  how  incorporating  testing  may  add  a  degree  of 
‘seriousness’  to  course  material. 

Participants  also  discussed  the  idea  that  feedback  may  not  always  be  beneficial.  The  struggle  for 
trainees  to  understand  their  own  mistakes  may  be  required  in  order  to  recognize  and  overcome 
problematic  behaviour.  In  addition,  if  feedback  is  provided  the  wrong  way,  it  can  be  ineffective. 
For  instance,  delays  in  receiving  feedback  may  cause  the  trainee  to  assume  the  worst  about  their 
performance  and  unnecessarily  diminish  their  confidence. 

Further,  participants  voiced  their  expectations  that  the  type  of  feedback  and  testing  would  impact 
on  learning  and  retention.  The  use  of  multiple  choice  tests  was  not  thought  to  be  the  most 
effective  method  of  assessing  learned  material,  and  participants  argued  that  this  method  of  testing 
may  not  maximize  the  learning  experience.  Flowever,  responsibility  for  developing  and 
administering  labour  intensive  feedback  and  testing  (i.e.,  long  answer  questions  versus  multiple- 
choice  questions)  would  fall  on  the  instructor.  Participants  expressed  concern,  then,  that  this  type 
of  ‘better’  feedback  and  testing  would  negatively  affect  instructor’s  responsibilities,  resources, 
and  workload. 

Research  investigating  effects  of  feedback  and  testing  could  focus  on  meta-cognition.  Participants 
suggested  that  research  should  explore  whether  helping  learners  to  observe  their  own 
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performance  and  provide  their  own  feedback  (e.g.,  self-correcting  their  mistakes)  might  help  them 
to  excel.  Research  questions  that  emerged  from  the  discussion  and  participant  comments  are 
noted  in  Table  22. 


Table  22  Feedback  and  testing  -  Research  questions 


Research  Topic 

Research  Questions 

Feedback  and  Testing 

How  can  feedback  be  implemented  to  optimize  and  accelerate  learning  and 
retention? 

How  can  testing  be  implemented  to  optimize  and  accelerate  learning  and 
retention? 

Can  effective  feedback  and  testing  be  developed  and  implemented  with 
minimal  resources  allotted  to  the  instructor? 

How  can  meta-cognition  be  best  applied  to  feedback  and  testing 

How  would  strategies  of  how  to  provide  their  own  feedback  affect  trainee 
learning  and  retention? 

3. 3. 2. 4.4  Adaptive  instruction 

Adaptive  instruction  can  be  defined  as  the  deliberate  tailoring  of  training  and  education  to  the 
needs  of  the  individual. 

Participants  noted  that  emerging  technologies  play  a  large  role  in  adaptive  instruction.  They 
understood  adaptive  instruction  in  terms  of  an  instruction  tool  rather  than  as  individual  learning, 
which  may  be  more  in  line  with  CF  training  goals.  Concerns  were  raised  that  implementation  of 
adaptive  instruction  may  be  limited  by  a  number  of  factors. 

Improvement  and  development  of  today’s  technology,  such  as  the  use  of  simulations,  may 
increase  the  benefit  of  adaptive  instruction  in  terms  of  promoting  accelerated  learning. 
Participants  argued  that  although  not  previously  feasible  before  the  introduction  of  technology, 
adaptive  instruction  can  be  used  within  multi-player  simulations.  In  these,  there  is  no 
predetermined  end-state  and  the  ‘game’  is  meant  to  evolve  based  on  player  activities  in  order  to 
promote  decision-making  and  team  work  skills. 

For  some  participants,  adaptive  instruction  was  seen  to  be  more  relevant  at  the  instructor  level 
rather  than  for  using  technology  to  provide  tailoring  to  individual  needs.  For  instance,  participants 
discussed  adaptive  instruction  as  a  product  evolved  from  AARs  and  lessons  learned.  That  is, 
training  is  adapted  based  on  lessons  learned  rather  than  on  the  difficulties  of  specific  individuals. 
At  the  same  time,  however,  participants  recognized  that  this  type  of  adaptive  instruction  may 
benefit  skill  decay.  Flowever,  some  participants  argued  that  there  may  not  be  any  prevalent 
evidence  that  such  adaptive  instruction  actually  works. 

Concerns  were  raised  that  adaptive  instruction  may  be  hindered  by  large  class  sizes,  scheduling, 
and  instructor  workload.  Further  research  may  focus  on  what  technologies  are  available  to  assist 
in  developing  and  implementing  adaptive  instruction  within  systemic  limitations  (e.g.,  resources, 
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class  size,  scheduling,  timing,  instructor  resources).  Participants  were  unsure  about  how  and 
whether  learning  can  be  tailored  to  individual  needs  in  order  to  promote  accelerate  learning. 


Research  questions  that  emerged  from  the  discussion  and  participant  comments  are  noted  in  Table 
23 


Table  23  Adaptive  instruction  -  Research  questions 


Research  Topic 

Research  Questions 

Adaptive  Instruction 

What  new  technologies  are  available  for  adaptive  learning?  Specifically  in 
multi-player,  collaborative  simulations? 

Would  it  be  most  beneficial  to  leverage  adaptive  training  as  a  tool  for 
instruction  rather  than  specific  to  the  individual? 

What  technologies  are  available  to  assist  in  developing  and  implementing 
adaptive  instruction  within  systemic  limitations  (e.g.,  resources,  class  size, 
scheduling,  timing,  instructor  resources)? 

How  do  you  adapt  to  the  responses  of  the  learner?  Would  that  be  helpful? 

3. 3.2.5  Context  and  content 

The  context  and  content  category  includes  other  features  of  the  environment  within  which 
learning  occurs,  as  well  as  specific  contents  of  training  (e.g.,  overlearning,  whether  cross-training 
occurs) 

In  general,  participants  discussed  the  difficulty  of  rating  the  category  of  context  and  content 
because  the  category  contained  multiple  ideas.  Participants  noted  that  the  context  and  content  of 
training  are  essential  in  understanding  learning  and  retention,  though  there  was  no  further 
elaboration.  Information  flow,  shift  training,  and  levels  of  stress  during  training  were  noted 
during  the  discussion. 

Information  flow  was  noted  by  multiple  groups  and  was  a  topic  also  discussed  in  collaborative 
learning.  As  a  general  observation,  participants  noted  the  CF  requires  a  more  flexible  system  that 
packages  and  relays  lessons  learned  (e.g.,  new  or  changed  techniques,  tactics  and  procedures)  to 
troops  on  the  ground  within  reasonable  time  in  order  to  increase  operational  effectiveness  and 
success. 

Participants  discussed  day  versus  night  training  and  testing  and  how  this  may  affect  learning  and 
retention.  Day/night  shift-training  has  been  implemented  in  simulated  environments,  and  is  often 
seen  as  an  efficient  use  of  simulation.  The  effects  on  learning  and  retention  however,  have  not 
been  researched. 

The  level  of  stress  that  should  be  induced  during  training  is  another  factor  that  can  significantly 
impact  training.  Participants  noted  the  extent  to  which  stress  should  be  implemented  during 
training  is  not  clear. 

Research  questions  that  emerged  from  the  discussion  and  participant  comments  are  noted  in  Table 
24. 
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Table  24  Context  and  content  -  Research  questions 


Research  Topic 

Research  Questions 

Context  and  Content 

How  can  the  CF  streamline  learning  and  circulate  lessons  learned  more 
quickly? 

How  can  the  CF  cut  down  on  middle  steps  (data  reporting,  meetings)  in  order  to 
hasten  the  circulation  of  lessons  learned? 

How  does  shift  training  affect  learning  and  retention? 

How  can  stress  be  used  to  accelerate  learning  and  increase  retention? 

Is  it  productive  or  counterproductive  to  induce  high  levels  of  stress  during 
training  (e.g.,  as  is  done  in  MARS  training)? 

3. 3. 2. 6  Retention 

There  were  three  separate  topics  related  to  retention  discussed.  These  are  expanded  upon  below. 

3. 3. 2. 6.1  Retention  -  Relationship  to  learning  factors 

Retention  is  defined  in  the  literature  as  the  maintenance  or  sustainment  of  learned  behaviours 
without  practice  or  “the  degree  of  competence  to  which  an  acquired  skill  [or  knowledge]  is 
retained  through  the  passage  of  time”  (Ginzburg  &  Dar-El,  2000,  p.  327). 

During  the  workshop  discussion,  topics  identified  under  learning  (student  attributes,  instructional 
learning,  delivery  media,  delivery  methods)  were  not  explicitly  discussed  in  relation  to  retention. 
Instead,  the  discussion  concerning  this  topic  focused  on  general  ideas  of  research,  level  of 
training,  and  transfer  of  learned  material. 

There  was  some  agreement  among  participants  that  research  in  this  field  may  be  out-dated,  and/or 
not  currently  effectively  applied.  One  participant  suggested  that  the  level  of  difficulty  in  training 
may  reduce  test  scores  and  seemingly  reduce  level  of  learning,  but  perhaps  retention  is 
maintained  or  even  increased. 

The  issue  of  transfer  came  up  in  the  discussion  connecting  learning  and  retention.  Participants 
argued  that  there  may  be  training  in  which  trainees  perform  well,  but  because  conditions  of  the 
environment  are  so  different  when  they  get  out  into  the  field,  little  of  the  training  is  transferred. 
On  a  similar  note,  there  may  be  a  disconnect  of  transferring  learned  material  if  trainees  are 
motivated  during  a  training  program  but  are  not  motivated  to  apply  the  learned  material  in  the 
field. 

Research  questions  that  emerged  from  the  discussion  and  participant  comments  are  noted  in  Table 
25 


Table  25  Relationship  to  learning  factors  -  Research  questions 


Research  Topic 

Research  Questions 

Retention:  Relationship 

What  areas  of  research  in  retention  are  out-dated?  (this  may  consider  effects 
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Research  Topic 

Research  Questions 

to  Learning  Factors 

of  new  technologies) 

How  might  the  details  of  a  training  program  (e.g.,  level  of  difficulty, 
scheduling,  day  vs.  night  training)  affect  retention? 

What  student  attributes  might  affect  retention  (e.g.,  motivation)? 

How  learned  knowledge  is  best  transferred? 

What  factors  might  affect  the  transfer  of  learned  material?  How  do  these 
factors  affect  transfer  of  learned  material  as  reflected  in  testing  and 
performance? 

3. 3. 2. 6.2  Models  of  knowledge/skill  retention 

Discussion  on  models  of  knowledge  and  skill  retention  focused  on  the  idea  of  refresher  training, 
the  type  of  models  that  may  be  useful  to  the  CF,  and  the  impact  of  technology. 

In  multiple  groups,  participants  discussed  that  skill  fading  and  refresher  training  is  done 
seemingly  based  on  arbitrary  industry  standards  rather  than  on  military-based  research.  General 
research  in  retention  may  also  be  very  out-dated  and  may  overlook  today’s  standards  and 
technologies. 

In  the  CF,  scheduling  of  resources  seems  to  be  the  main  factor  that  dictates  when  refresher 
training  is  implemented.  Participants  suggested  that  working  to  create  models  investigating  when 
refresher  training  is  required  and  how  best  it  can  be  implemented  given  the  systematic  constraints 
of  the  CF  would  be  an  important  research  priority. 

Multiple  participants  argued  that  developing  a  model  of  retention  that  is  experiential  will  have 
more  relevance  to  the  CF  than  a  mathematical  model.  It  is  expected  that  the  number  of  variables 
included  in  such  a  model  would  be  extensive.  For  example,  the  boundary  of  learning  and 
retention  could  include  all  formal  training,  informal  learning,  and  collaborative  learning.  An 
investigation  and  validation  of  existing  models  may  prove  useful  in  developing  an  approach  to 
improve  retention  within  the  CF.  Information  development  occurs  much  faster  today  than  it  has  in 
the  past.  The  speed  of  learning  has  completely  changed  and  current  models  of  knowledge  and 
skill  decay  may  be  obsolete. 

Research  questions  that  emerged  from  the  discussion  and  participant  comments  are  noted  in  Table 
26. 


Table  26  Models  of  knowledge/skill  retention  -  Research  questions 


Research  Topic 

Research  Questions 

Models  of 

Knowledge  /  Skill 
Retention 

When  is  refresher  training  required? 

How  is  refresher  training  best  implemented  given  the  systematic  constraints  of 
the  CF? 

What  type  of  experiential  model  of  retention  may  benefit  training  within  the  CF? 
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3. 3. 2. 6.3  Techniques  for  retention 


Participants  noted  that  it  may  be  useful  to  look  to  other  industries  for  research  on  techniques  for 
retention.  The  use  of  job  aids  were  also  discussed  as  a  technique  to  increase  retention  resulting  in 
increased  performance. 

There  may  be  extensive  literature  within  the  corporate  training/industrial  world  that  could  be 
useful  to  the  CF.  Options  for  further  action  may  be  simply  to  reconnect  with  the 
corporate/business  world  and  understand  techniques  for  retention  from  their  perspective. 

in  order  to  effectively  perform,  it  may  be  preferable  to  provide  troops  with  job  aids.  Often, 
trainees  do  not  have  access  to  critical  material  after  the  course  and  do  not  have  an  opportunity  to 
refresh  their  learning  unless  it  is  within  the  context  of  a  structured  course  (e.g.,  refresher  training). 
CF  personnel  may  benefit  from  investigation  of  using  job  aids  (e.g.,  a  little  book  with  key  terms 
and  information)  which  would  refresh  the  material  as  needed.  This  may  result  in  increased 
performance. 

Research  questions  that  emerged  from  the  discussion  and  participant  comments  are  noted  in  Table 
27. 


Table  27  Techniques  for  retention  -  Research  questions 


Research  Topic 

Research  Questions 

Techniques  for 
retention 

How  can  literature  from  industry  be  leveraged  to  identify  and  implement 
techniques  for  retention  within  the  CF? 

Can  job  aids  be  implemented  within  the  CF  to  assist  in  maintaining  high 
retention  rates  and  increase  performance? 

3.3.3  Quantitative  Results 
3. 3.3.1  Ratings  of  Topics 

During  the  workshop  summary,  preliminary  analysis  of  the  matrices  data  was  undertaken  by 
CDA  facilitators.  Based  on  the  entries  of  participants  (and  participants’  additive  totals  of  the 
numbers),  this  analysis  showed  simulation-based  instruction;  instructor  attributes,  collaborative 
learning,  and  distance  learning  among  the  top  four.  Distance  learning  was  ranked  as  the  number  1 
priority  in  this  quick  preliminary  analysis.1 

Upon  presentation  of  these  preliminary  rankings  during  the  after  action  review  (AAR),  however, 
participants  expressed  surprise  that  distance  learning  seemed  to  have  emerged  as  the  most 
important  topic.  One  participant  noted  that  their  group  had  dismissed  distance  learning  as  an 
important  topic  because  of  the  vast  amount  of  associated  research  currently  available.  Others 


1  The  rank  discrepancy  from  the  preliminary  analysis  was  due  to  incorrect  addition  on  the  part  of  several  participants. 
This  error  was  corrected  during  the  full  analysis. 
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pointed  out  that  distance  learning  can  be  considered  a  combination  of  a  number  of  other  issues 
(e.g.,  computer-based  instruction,  scheduling). 


The  fact  that  participants  had  expressed  some  surprise  about  distance  learning  being  the  most 
important  research  priority  came  into  a  clearer  light  with  more  detailed  data  analyses.  With  the 
time  available  to  use  the  full  data  set  provided  by  participants  (rather  than  relying  on  their  totals), 
it  became  clear  that  some  of  the  preliminary  calculations  were  incorrect.  In  actuality,  more 
detailed  analyses  showed  a  somewhat  different  pattern. 

Table  28  shows  the  mean  importance  ratings  of  each  topic  (and  associated  standard  deviations) 
for  each  of  the  13  participants. 


Table  28  Rating  and  rank  of  topics 


Topic 

N 

(sample 

size) 

Mean 

Standard 

Deviation 

Rank 

Simulator-based  instruction 

13 

3.99 

0.42 

1 

Instructor  attributes 

13 

3.8 

0.59 

2 

Collaborative  learning 

13 

3.8 

0.55 

3 

Distance  learning 

13 

3.74 

0.48 

4 

Computer-based  Instruction 

13 

3.74 

0.69 

5 

Techniques  for  retention 

13 

3.7 

0.62 

6 

Feedback  and  testing 

13 

3.69 

0.37 

7 

Embedded  training 

13 

3.69 

0.57 

8 

Context  and  content 

13 

3.66 

0.56 

9 

Models  of  knowledge  /skill  retention 

13 

3.65 

0.65 

10 

Student  attributes 

13 

3.59 

0.85 

11 

Adaptive  instruction 

13 

3.58 

0.63 

12 

Retention-Relationships  to  learning  factors 

11 

3.52 

0.81 

13 

Classroom  aids /  environment 

13 

3.45 

0.42 

14 

Scheduling 

12 

3.22 

0.73 

15 

Mean 

3.66 

0.6 

Based  on  the  mean  importance  rating  they  received,  the  15  topics  could  then  be  empirically 
ranked  in  order  of  relative  importance,  treating  all  the  criteria  as  equal.  The  top  ranked  topic  is 
Simulator-based  instruction  with  a  mean  of  3.99,  followed  by  Instructor  Attributes  and 
Collaborative  Learning  tied  in  second  place  with  a  mean  of  3.80.  This  was  closely  followed  by 
Distance  learning  and  Computer-based  instruction  (also  tied)  with  a  mean  of  3.74. 

The  fact  that  the  ratings  of  all  topics  are  above  the  midpoint  of  the  scale  (i.e.,  3)  suggests  that 
participants  saw  all  of  the  topics  as  making  a  meaningful  contribution  to  future  research  and 
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development  efforts.  Even  the  lowest  rated  topic  (scheduling  with  the  mean  of  3.22)  received  a 
rating  above  the  midpoint  of  the  rating  scale. 

Given  the  low  sample  size,  extensive  analyses  were  not  possible.  However,  given  that  the  ratings 
were  provided  by  all  participants,  it  was  possible  to  explore  whether  the  means  of  the  different 
topics  (ranging  from  3.99  for  simulation-based  instruction  to  3.22  for  scheduling)  were  actually 
statistically  different.  This  analysis  showed  that  the  importance  ratings  for  all  15  topics 
simultaneously  did  not  differ  significantly,  although  the  highest  and  the  lowest  rated  topics  were 
significantly  different. 

Additional  analyses  were  undertaken  to  explore  the  topics  in  relation  to  the  1 1  evaluation  criteria. 
Given  the  pattern  of  results,  it  was  important  to  understand  why  people  believed  simulator-based 
instruction  was  the  highest  priority.  As  shown  in  Annex  E,  simulation-based  instruction  was  rated 
as  having  the  highest  level  of  scientific  merit  (m  =  4.54),  as  being  the  most  aligned  with  and 
supportive  of  the  goals  of  the  CF  (4.38),  and  as  being  the  most  likely  to  attract  industrial 
stakeholders  (m  =  4.62).  Importantly,  however,  it  was  also  rated  as  being  in  the  mid-range  of  ease 
of  implementation.  This  suggests  that  even  those  participants  saw  it  to  be  highly  important,  they 
were  less  confident  about  the  ability  of  the  CF/DND  to  implement  a  wide  range  of  simulation- 
based  instruction. 

3. 3. 3. 2  Ranking  of  Topics 

The  matrix  spreadsheet  included  an  additional  column  in  which  participants  were  expected  to 
provide  rankings  of  the  relative  importance  of  the  topics  based  on  a  more  holistic  evaluation.  This 
would  offer  a  different  perspective  from  the  criteria  ratings.  Unfortunately,  only  one  of  the  3 
break-out  groups  received  specific  instructions  to  complete  these  rankings  and  information  about 
how  to  do  this.  This  resulted  in  only  four  participants  ranking  all  topics,  and  one  participant 
ranked  only  their  top  three  topics. 

3. 3. 3. 3  Evaluation  Criteria 

It  was  also  important  to  explore  how  participants  rated  the  topics  as  a  whole  on  the  evaluation 
criteria.  This  would  provide  a  broad  perspective  on  whether  the  15  topics  identified  during  the 
literature  review  were  actually  seen  to  be  important  in  the  eyes  of  the  workshop  participants. 
Means  and  standard  deviations  for  each  criterion  are  shown  in  Table  29. 


Table  29  Means  (standard  deviations)  of  evaluation  criteria 


N  (sample 
size) 

Mean 

Standard 

Deviation 

A  -  The  topic  has  scientific  merit. 

189 

3.98 

1.00 

B  -  The  topic  is  novel  or  innovative  in  addressing  learning 

190 

3.48 

1.18 

C  -  The  topic  aligns  with  and  supports  CF/DND  goals. 

179 

3.91 

0.96 

D  -  The  topic  will  have  wide  application  within  CF/DND. 

186 

3.95 

1.01 

E  -  The  topic  will  improve  organizational  efficiency  and/or 

181 

3.88 

0.91 
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F  -  The  topic  will  address  identified  gaps  -  learning  and 
retention  deficits  -  in  the  CF/DND. 

167 

3.47 

0.96 

G  -  The  anticipated  results  of  addressing  the  topic  will 
accelerate  learning  and/or  improve  retention. 

185 

3.72 

0.96 

H  -  The  anticipated  results  will  be  relatively  easy  to 
implement. 

178 

2.80 

1.11 

1  -  National  or  international  collaboration  on  this  topic  is 
possible. 

171 

3.85 

1.06 

J  -  The  topic  will  attract  an  industrial  stakeholder  for  R&D. 

150 

3.33 

1.26 

K  -  There  is  sufficient  capability  and/or  capacity  within 
Canada  to  take  on  this  topic  for  R&D. 

153 

3.78 

0.93 

As  this  table  shows,  then,  participants  agreed  that  the  1 5  topics  had  a  high  level  of  scientific  merit 
(mean  =  3.98),  that  the  results  of  emerging  research  would  have  wide  application  within  CF/DND 
(mean  =  3.95),  and  that  they  were  aligned  and  congruent  with  CF/DND  goals  (mean  =  3.91). 
Perhaps  not  surprisingly,  the  criterion  with  the  lowest  rating  addressed  how  easy  it  would  be  to 
implement  the  findings  of  research.  Given  some  of  the  break-out  group  discussions,  this  rating 
might  reflect  that  even  accumulating  the  best  possible  research  and  development  may  not  address 
pervasive  resource  limitations  of  military  systems.  Ratings  for  all  other  evaluation  criteria  were 
above  the  midpoint  of  the  scale,  reflecting  good  agreement  about  the  importance  of  future 
research  within  these  topic  areas,  and  good  potential  for  meaningful  contributions  to  the 
CF/DND. 

3.3.4  After  Action  Review 

As  a  conclusion  to  the  workshop,  participants  were  brought  together  to  discuss  their  general 
thoughts  on  the  topics,  their  ratings,  and  overall  critique  of  the  workshop.  The  final  group 
discussion  was  intended  to  receive  participant  feedback  about  their  completion  of  the  matrix,  and 
about  their  experience  in  the  workshop. 

Topics  -  After  discussion  during  the  break-out  sessions,  participants  had  completed  the  matrix 
rating  each  of  the  15  topics  on  the  evaluation  criteria.  One  challenge  noted  by  participants, 
however,  was  that  some  topics  were  multidimensional,  but  required  a  single  rating.  This  meant 
that  their  rating  of  that  topic  would  depend  on  how  they  constructed  a  single  rating.  A  few 
participants  noted  that  their  ratings  would  change  if  they  had  the  ability  to  narrow  the  topic  to  a 
specific  idea.  Although  it  is  unclear  how  participants  made  this  determination,  it  seems  unlikely 
that  they  used  the  same  strategy. 

Several  participants  noted  specifically  that  the  Scheduling  and  Context  and  Content  topics  were 
too  broad,  consequently  asterisks  were  marked  for  the  majority  (or  all)  of  these  criteria  for  these 
participants.  Participants  noted  that  even  with  the  definition  provided  from  the  PowerPoint 
presentation,  they  felt  the  topics  were  too  broad  to  accurately  assess  (Scheduling,  Relationships  to 
Learning  Factors),  or  terminology  was  unclear  (Embedded  Training). 

Participants  suggested  that  criteria  or  topics  that  with  high  levels  of  non-response  (i.e.,  not  rated, 
but  instead  marked  by  at  *)  should  be  re-examined.  For  example,  as  described  earlier,  topics 
related  to  Scheduling  and  to  the  Relationship  between  learning  factors  and  retention  factors  were 
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not  well  understood  by  some  participants  who  used  asterisks  to  rate  all  of  the  criteria  for  these 
topics. 

Evaluation  Criteria  -  In  further  discussion,  participants  noted  that  some  criteria  may  be  more 
important  than  others  and  heavier  weighting  should  be  given  to  those  criteria.  This  would  change 
the  average  ratings,  and  it  would  be  interesting  to  then  see  how  participant  rankings  compare. 

Specifically,  participants  judged  the  last  three  criteria  as  being  particularly  difficult  to  judge. 

1  -  National  or  international  collaboration  on  this  topic  is  possible; 

J  -  The  topic  will  attract  an  industrial  stakeholder  for  R&D;  and 

K  -  There  is  sufficient  capability  and/or  capacity  within  Canada  to  take  on  this  topic  for 
R&D 

Based  on  this  difficulty,  they  argued  that  these  criteria  should  be  given  less  weight  when 
considering  the  results  of  participant  ratings.  Unfortunately,  missing  data  does  not  allow  for  these 
analyses. 

General  Discussion  -  Participants  voiced  concern  that  although  there  may  be  clear  benefits  of 
implementing  some  changes  to  training  in  the  long  term,  it  may  not  be  feasible  within  the 
structure  of  the  CF.  Hence,  some  suggested  that  the  feasibility  of  implementation  of  research 
outcomes  should  be  weighted  significantly  higher  when  considering  future  research  and 
development. 

Research  areas  will  not  be  initiated  if  feasibility  within  the  CF  is  low.  Designating  further 
research  areas  will  require  a  balance  between  research  needs  identified,  value,  feasibility,  and  CF 
priorities. 

A  group  discussion  confirmed  that  participants  felt  the  workshop  was  an  interesting  and  effective 
way  to  elicit  knowledge  from  stakeholders.  Several  suggestions  relevant  to  future  workshops  also 
emerged.  These  included  using  mind  maps  to  ensure  common  understanding  about  the  meaning 
of  the  topic  before  break-out  groups  and/or  allowing  groups  to  create  their  own  mind  maps.  This 
would  ensure  a  clear  idea  of  what  the  topic  included. 

Suggestions  also  included  accessing  the  expertise  of  stakeholders  in  subsequent  workshops 
through  their  ratings  and  rankings  of  more  specific  research  questions.  As  noted  earlier,  some 
participants  argued  that  the  topics  explored  within  this  workshop  were  sometimes  too  broad. 


3.4  Overview  of  workshop  results 

Participant  discussions  and  their  comments  on  the  rating  matrix  provided  a  range  of  research 
questions  about  each  of  the  15  topics,  as  compiled  in  Table  30 


Table  30  Research  questions  gathered  from  discussion  groups  (in  order  of presentation) 


Research  Topic 


Research  Questions 
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Research  Topic 

Research  Questions 

Student  attributes 

•  Can  cultural  differences  be  utilized  to  accelerate  certain  types  of  training? 

•  Does  multitasking  improve  overall  productivity  for  the  new  generation? 

o  If  informal  learning  takes  place  in  multitasking  environments,  what 
are  the  effects  on  retention? 

•  What  causes  high  motivation?  Is  it  the  student’s  intrinsic  motivation,  influence 
of  the  instructor,  or  classroom  aids 

•  How  does  accounting  for  student  attributes  in  practice  schedules  influence 
training  effectiveness? 

Instructor  attributes 

•  What  are  effective  instructor  attributes  and  how  can  they  be  trained? 

•  How  might  the  selection  of  instructors  be  influenced  based  on  effective 
instructor  attributes? 

•  Can  the  context  or  of  instruction  (class,  distance,  mentoring)  be  matched  to 
instructor  preferences  and  capabilities  in  order  to  accelerate  training? 

•  How  does  operational  currency  (or  lack  thereof)  influence  the  ability  to 
effectively  train  students?  Could  improving  operational  currency  be 
incorporated  into  a  training  program  in  a  fashion  that  would  accelerate  learning 
and/or  improve  retention? 

Classroom  aids  / 
Environment 

What  aids  currently  exist  in  civil  society  that  effectively  accelerate  learning? 

Can  these  aids  be  feasibly  implemented  within  the  current  CF  system  and  resources? 
What  makes  classroom  aids  effective  for  accelerated  learning? 

Is  it  the  ability  of  the  instructor  to  effectively  implement  it? 

Is  it  motivation  of  the  student  that  is  increased  by  its  use? 

Does  the  CF  have  classroom  aids?  If  so,  can  they  be  evaluated,  or  implemented 
differently  in  order  to  achieve  accelerated  learning  and  improve  retention. 

Distance  Learning 

How  does  one  design  a  distance  learning  course  with  a  superior  instructional  design  that 
can  accelerate  learning? 

If  courses  are  developed  with  a  superior  instructional  design,  can  they  be  successful  at 
accelerating  learning  while  having  the  additional  benefit  of  eliminating  travel  time? 

What  is  the  best  implementation  of  distance  learning? 

What  are  the  best  tests  for  distance  learning? 

How  are  distance  learning  tests  best  implemented? 

Are  there  better  ways  to  implement  distance  learning  given  new  tools  and  technology? 

Computer-based 

instruction 

What  computer-based  instruction  tools  exist  that  may  be  beneficial  to  the  CF? 

In  which  cases  is  self-paced  computer-based  instruction  beneficial? 

In  which  cases  is  computer-based  instruction  that  is  systematically  paced  beneficial? 

What  skills  can  most  effectively  be  trained  through  computer-based  instruction? 

Is  there  current  research  that  demonstrates  the  benefit  of  computer-based  instruction? 

Simulation-based 

instruction 

What  type  of  simulation  is  best  for  what  context? 

What  aspects  of  realism  are  lost  in  using  simulation-based  instruction?  Are  these 
essential  in  training?  If  so,  how  can  their  loss  be  mitigated? 
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Research  Topic 

Research  Questions 

Can  simulations  be  used  to  effectively  train  soft  skills? 

How  does  virtual  online  'realities’  of  simulations  effect  training,  performance,  learning, 
and  retention? 

Can  simulation  technology  be  more  flexible  /  versatile  in  a  variety  of  training? 

Can  training  curricula  be  more  effectively  developed  with  designers  and  trainers  are 
involved  in  multiple  stages  of  the  creation  and  implementation  of  simulation  based 
instruction? 

Embedded  Training 

How  can  embedded  training  be  utilized  within  the  CF? 

How  can  embedded  training  be  implemented  effectively  in  the  CF  to  achieve  accelerated 
learning  and/or  increase  retention? 

Collaborative 

Learning 

Should  military  teams  be  taught  to  learn  together  as  a  team  rather  than  simply  work 
together  as  a  team?  If  so,  how  is  this  best  accomplished? 

How  can  collaborative  and  team  learning  be  implemented  to  maximize  training 
experiences  for  all  trainees  within  the  limitations  of  resources  (e.g.,  time,  equipment)? 

Does  information  flow  effectively  throughout  the  CF  in  order  to  optimize  training?  How 
can  this  be  improved?  How  can  this  be  insured? 

What  are  the  social  aspects  of  collaborative  learning  that  may  affect  learning?  How  is  it 
best  to  consider  them  to  optimize  learning  and  retention? 

What  type  of  technology  (e.g.,  social  networking,  social  media)  plays  a  role  in 
collaborative  learning?  How  might  it  be  leveraged  for  training? 

Research  Topic 

Research  Questions 

Scheduling 

How  does  scheduling  effect  learning? 

How  can  the  sequencing  of  training  programs  be  optimized  to  lead  to  accelerated 
learning  and/or  improved  retention? 

Are  there  inefficiencies  within  the  current  system  that  negatively  impact  on  learning, 
retention  and  consequently  performance? 

Can  scheduling  be  improved  to  reduce  the  delay  between  training  and  application  of 
skills? 

How  can  scheduling  be  optimized  to  accelerate  learning  given  the  extensive  number  of 
opportunities  presented  through  technology? 

Feedback  and  Testing 

How  can  feedback  be  implemented  to  optimize  and  accelerate  learning  and  retention? 

How  can  testing  be  implemented  to  optimize  and  accelerate  learning  and  retention? 

Can  effective  feedback  and  testing  be  developed  and  implemented  with  minimal 
resources  allotted  to  the  instructor? 

How  can  meta-cognition  be  best  applied  to  feedback  and  testing 

How  would  strategies  of  how  to  provide  their  own  feedback  affect  trainee  learning  and 
retention? 

Adaptive  Instruction 

What  new  technologies  are  available  for  adaptive  learning?  Specifically  in  multi-player, 
collaborative  simulations? 

Would  it  be  most  beneficial  to  leverage  adaptive  training  as  a  tool  for  instruction  rather 
than  specific  to  the  individual? 

What  technologies  are  available  to  assist  in  developing  and  implementing  adaptive 
instruction  within  systemic  limitations  (e.g.,  resources,  class  size,  scheduling,  timing, 
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Research  Topic 

Research  Questions 

instructor  resources)? 

How  do  you  adapt  to  the  responses  of  the  learner?  Would  that  be  helpful? 

Context  and  Content 

How  can  the  CF  streamline  learning  and  circulate  lessons  learned  more  quickly? 

How  can  the  CF  cut  down  on  middle  steps  (data  reporting,  meetings)  in  order  to  hasten 
the  circulation  of  lessons  learned? 

How  does  shift  training  affect  learning  and  retention? 

How  can  stress  be  used  to  accelerate  learning  and  increase  retention? 

Is  it  productive  or  counterproductive  to  induce  high  levels  of  stress  during  training  (e.g., 
as  is  done  in  MARS  training)? 

Retention: 

Relationship  to 

Learning  Factors 

What  areas  of  research  in  retention  are  out-dated?  (this  may  consider  effects  of  new 
technologies) 

How  might  the  details  of  a  training  program  (e.g.,  level  of  difficulty,  scheduling,  day  vs. 
night  training)  affect  retention? 

What  student  attributes  might  affect  retention  (e.g.,  motivation)? 

How  is  learned  knowledge  best  transferred? 

What  factors  might  affect  the  transfer  of  learned  material?  How  do  these  factors  affect 
transfer  of  learned  material  as  reflected  in  testing  and  performance? 

Models  of  Knowledge 
/  Skill  Retention 

When  is  refresher  training  required? 

How  is  refresher  training  best  implemented  given  the  systematic  constraints  of  the  CF? 
What  type  of  experiential  model  of  retention  may  benefit  training  within  the  CF? 

Techniques  for 
retention 

How  can  literature  from  industry  be  leveraged  to  identify  and  implement  techniques  for 
retention  within  the  CF? 

Can  job  aids  be  implemented  within  the  CF  to  assist  in  maintaining  high  retention  rates 
and  increase  performance? 

Fifteen  concepts  were  presented  at  the  workshop  aimed  at  eliciting  feedback  from  stakeholders. 
Group  discussions  were  held  on  each  of  the  topics  and  participants  provided  research  questions, 
research  ideas,  comments,  and  completed  ratings  based  on  1 1  criteria  presented.  Four  participants 
also  rank  ordered  the  topics  in  addition  to  rating  them. 

Participant  ratings  of  the  importance  of  the  15  topics  showed  that  Simulator-based  instruction  was 
rated  as  the  highest  priority  with  a  mean  of  3.99,  followed  by  Instructor  Attributes  and 
Collaborative  Learning  tied  in  second  place  with  a  mean  of  3.80.  This  was  closely  followed  by 
Distance  learning  and  Computer-based  instruction  (also  tied)  with  a  mean  of  3.74.  All  topics  were 
rated  as  making  a  meaningful  contribution  to  future  research  and  development  efforts  as  average 
ratings  were  all  above  the  midpoint  of  the  scale. 

The  discussion  in  the  AAR  was  generally  consistent  with  the  overall  empirical  rankings.  That  is, 
the  use  of  technology  was  a  reoccurring  theme  throughout  the  majority  of  discussions.  There  was 
also  concern  of  the  feasibility  of  implementing  change  within  the  current  structure  CF  because 
resources  are  often  strained.  Overall,  the  participants  felt  that  the  workshop  was  a  productive 
activity  and  follow-on  work  also  involving  stakeholders  would  be  beneficial  to  the  research 
program. 
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As  a  whole,  then,  the  goals  of  workshop  were  met.  The  information  provided  by  participants 
offers  an  important  perspective  on  the  needs  of  the  CF/DND  system  as  it  works  to  conduct 
research  and  development  to  improve  its  training  and  education  system. 
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4  Conclusions 


The  purpose  of  this  work  was  to  identify  the  most  promising  topics  for  Research  and 
Development  (R&D)  on  Accelerating  Learning  and  maximizing  knowledge/skill  retention  that 
could  yield  greatest  benefit  to  the  Canadian  Forces  (CF)  within  five  (5)  years.  This  scoping  study 
consisted  of  a  review  of  relevant  literature  and  a  workshop.  The  literature  review  provided  a 
broad  overview  of  current  knowledge  in  the  scientific,  military,  and  related  literature  relevant  to 
accelerated  learning  and  retention.  It  was  noted  that  the  factors  that  influence  skill  acquisition  are 
not  necessarily  the  same  ones  that  influence  skill  retention.  The  literature  review  showed  that  the 
construct  of  accelerated  learning  is  still  at  a  relatively  early  stage  of  development  and  that  much 
research  is  required  to  fully  understand  accelerated  learning  and  retention  in  a  CF  context. 

As  noted  in  the  previous  section,  the  primary  outcome  of  the  workshop  was  the  rating  of  the 
primary  topics  of  interest.  Not  surprisingly,  simulator-based  instruction  received  the  highest 
importance  rating  from  participants,  and  was  seen  to  have  a  high  level  of  scientific  merit  as  well 
as  being  the  most  aligned  with  and  supportive  of  the  goals  of  the  CF,  and  as  being  the  most  likely 
to  attract  industrial  stakeholders.  Given  the  strong  push  to  use  simulation  for  training  within  both 
civilian  and  military  contexts,  this  result  is  perhaps  unsurprising.  Given  the  high  ratings  for 
distance  learning  and  computer-based  learning,  there  is  clear  agreement  about  the  potential  role 
that  technology  needs  to  play  in  helping  to  promote  accelerated  learning.  Knowing  more  about 
how  technology  can  be  used  to  promote  higher  levels  of  learning  and  retention  will  be  critical  as 
future  efforts  unfold. 


Even  among  all  the  new  technologies  that  could  be  used  to  promote  learning,  however,  it  is 
important  to  note  that  attributes  of  the  human  instructor  received  the  second  highest  rating  from 
workshop  participants.  They  noted  the  importance  of  research  exploring  the  match  between 
instructor’s  capabilities  and  preferences,  and  the  new  technologies  used  for  instruction  such  as 
distance  education.  What  makes  an  instructor  capable  of  being  effective  even  when  interacting 
remotely  with  a  student  seems  an  especially  interesting  question  given  the  introduction  of  new 
learning  modalities. 


Given  the  nature  of  many  military  tasks,  collaborative  learning  is  another  topic  emerging  from 
this  project  that  should  receive  research  attention.  Understanding  how  team  members  can  best 
learn  what  they  need  to  learn  as  a  team  is  critical  to  ensure  the  highest  level  of  effectiveness. 
What  role  technology  could  play  in  facilitating  that  team  effectiveness  should  also  be  explored. 
Specifically,  there  is  interest  in  determining  if  popular  collaborative  technology  (e.g.,  social 
networking,  social  media)  could  be  effectively  used  in  CF  training. 


One  important  finding  emerging  from  both  the  literature  review  and  from  workshop  discussions  is 
that  there  is  no  single  topic  that  will  help  to  “take  the  brakes  off’  of  learning  and  promote  high 
levels  of  retention.  Indeed,  the  many  different  influences  on  learning  include  the  student,  the 
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instructor,  how  training  material  is  presented,  and  whether  it  occurs  collaboratively  or 
individually.  Nonetheless,  as  the  literature  review  shows,  although  there  is  a  complex  matrix  of 
influences  to  navigate,  there  is  also  considerable  potential  for  making  contributions  to  the 
literature  through  future  research  and  development  efforts. 
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5  Future  Research 


Focusing  on  the  top  six  ranked  concepts  arising  from  the  workshop  and  review  (i.e.,  simulator- 
based  instruction,  instructor  attributes,  collaborative  learning,  distance  learning,  computer-based 
instruction,  and  techniques  for  retention),  a  number  of  specific  future  research  options  are 
explored  below.  Specific  research  programs  could  focus  on  the  following  issues: 

♦  One  potential  means  of  encouraging  collaborative  learning  is  through  the  use  of 
social  networking  tools  (e.g.,  Facebook,  Twitter,  Blackboard,  etc.).  Specifically, 
research  could  explore  the  role  that  these  tools  might  have  in  facilitating  accelerated 
learning  in  the  CF.  This  research  could  document  current  social  networking  tool 
usage  within  the  CF  (if  any)  and  best  practices.  This  work  could  also  explore 
potential  uses  and  benefits  that  these  tools  might  offer.  This  could  be  facilitated  by 
exploring  how  these  tools  are  used  in  other  educational  (or  broader)  contexts.  This 
research  could  include  both  a  focus  on  how  the  tools  could  be  used  by  instructors 
(e.g.,  using  a  Wiki  page  to  collect  information  related  to  challenging  topic  areas)  as 
well  as  how  the  tools  could  be  used  by  students  (e.g.,  communicating  over  instant 
messenger  or  Twitter  for  project  collaboration). 

♦  The  role  of  changing  demographics  (i.e.,  aging  population,  generational  differences) 
on  education  and  training  in  the  CF  is  another  future  research  area.  Research  could 
begin  by  documenting  and  understanding  generational  differences  (e.g.,  acceptance 
of  technologies,  effects  of  multitasking  on  productivity)  and  the  impact  of  these 
differences  on  learning  and  retention.  Experimentation  could  also  be  conducted 
exploring  the  roles  that  these  factors  play  in  accelerated  learning  for  different 
demographics,  perhaps  suggesting  best  practices  based  on  learner  attributes. 

♦  More  of  a  research  focus  could  also  be  placed  on  trainers  in  order  to  develop  train- 
the-trainers  programs  that  can  lead  to  accelerated  learning.  This  program  could 
update  trainers  on  what  other  trainers  are  doing,  suggest  broader  training  options  and 
in  general  seek  to  ensure  that  trainers  are  up-to-date  on  established  best  practises  in 
education  and  training.  These  train-the -trainer  programs  could  be  developed  for 
specific  trainers  deemed  most  in  need,  or  more  generally  created  to  apply  to  all 
educators  and  trainers  in  the  CF. 

♦  As  discussed  in  earlier  sections  of  this  report,  the  full  list  of  15  potential  research 
concepts  were  all  rated  above  the  mean,  indicating  they  were  all  seen  as  relevant 
concepts  for  further  research  (see  Table  28).  Although  this  is  encouraging,  there  is 
little  information  in  these  ratings  that  helps  to  truly  distinguish  the  most  critical  CF 
needs.  With  that  in  mind,  it  may  be  beneficial  to  further  refine  the  current  rankings 
with  a  broader  set  of  expertise.  One  means  of  refinement  could  be  through 
interviews  at  various  educational  and  training  facilities  across  the  CF.  By  talking 
with  current  students  and  trainers  about  their  concerns  and  issues,  a  fuller 
understanding  of  the  major  impediments  to  accelerated  learning  could  be  reached. 
Analysis  of  these  interviews  could  further  inform  research  and  development  efforts 
seeking  to  benefit  training  within  the  CF  to  help  promote  acceleration  of  learning 
and  improved  retention. 
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Whatever  the  future  approach,  there  are  many  critical  questions  related  to  accelerated  learning 
and  retention  that  are  remaining  to  be  answered  through  future  research  and  development  efforts. 
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A  Project  on  Accelerated  Learning 
and  Skill  /  Knowledge  Retention 


Personnel  shortage 
Potential  generational  mismatch 
Increasing  job  complexity 
Stressors 

Training  policy  and  process 
Science  and  Technology  pull 


The  Science  of  this  project  is  study  of 

•  human  learning 

•  training  transfer,  and 

•  knowledge  and  skill  retention 

•  expertise  and  the  transition  of  novice  to  expert 

•  human  information  processing,  human  memory 
and  decision  making, 

•  all  the  factors  that  impact  on  these  abilities. 


Canadian  Defence  Academy 
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Learning,  training  transfer,  and  retention  are  not 
exclusive  to  human  performance 


•  Maybe  the  best  way  of  solving  these  problems 
will  be  a  merging  of  human  and  machine 
capabilities 
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•  Phase  1  -  a  scoping  study  and  workshop  that  will 

-  identify  the  factors  that  impact  most  upon  the  acquisition  and 
retention  of  knowledge  and  skill,  and 

-  identify  the  areas  for  R&D  that  will  have  most  impact  potential  by 
considering  the  technology  gaps,  national  S&T  capabilities,  and 
collaborative  opportunities 

•  Phase  2  -  a  project  plan  that  consists  of 

-  a  number  of  work  elements  that  address  the  target  areas 

-  contract  for  prosecution  of  the  work 

-  partnerships  and  collaborative  arrangements  for  leverage  of  the 
investments 

A  DRDC  /  CDA  work  team 

•  engaging  TTCP  countries,  through  HUM  TP2  (Training  Technology)  and 
to  ieveragethe  NATO  RTO 

•  follow-on  NATO  Exploratory  Team  (ET)  and  Research  Task  Group 
(RTG)toHFM  121/RTG1 65 

•  ADL  network  -  in  Canada  and  internationally 
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Workshop  Schedule  /  Process 


Time  Event 


0800 


0815 


0845 


0915 


1030 


1045 


1115 


1200 


1300 


1530 


Introduction 

LCdr  Peter  Ball  (Learning  Concepts,  CDAHQ,  Kingston  ON) 


Presentation 

Dr  Dee  Andrews  -  (Senior  Scientist,  Human  Effectiveness  Directorate, 
USAF  Research  Lab,  Mesa,  AZ  ) 


Presentation 

Dr  Robert  Hoffman  -  (Senior  Research  Scientist,  Florida  Institute  for  Human  and 
Machine  Cognition,  Pensacola,  FL) 


Topics 

Dr.  Barbara  Adams  (Senior  Human  Factors  Consultant,  HSI  Inc,  Guelph  ON) 


Refreshment  Break 


Criteria 

DrLochlan  Magee  (Defence  Scientist,  DRDC  Toronto  ON) 


Dry  run 

Capt  John  Wyville  (TESM  Standards,  CDA Headquarters,  Kingston  ON) 


Lunch  Break 


Break-out  Group  Activity 


Workshop  Summary 


Canadian  Defence  Academy 
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Annex  B  Guest  speaker  presentation:  Dr.  Dee  Andrews 


Accelerated  Learning  and 
Long  Term  Retention  of 
Expertise 

Feb  2011 

Dr.  Dee  Andrews 
Senior  Scientist 
Human  Effectiveness  Directorate 
Air  Force  Research  Laboratory 


Tasking  &  Definition 

—  DSTAG  Tasking  — 

“The  Human  Systems  team  should  baseline  the  existing  programs 
(understand  current/planned  investments  including  cognitive 
techniques)  and  identify  new  ideas/technologies  to  be 

pursued.” 

What  is  “accelerated  learning”  ? 

—  The  TFT’s  Definition  -- 

Any  learning  system  or  environment  that  attempts  to  control  for  time  spent  versus 
content  learned  with  the  following  goals: 

•  Faster  attainment  of  skill  and  knowledge  then  a  current  baseline,  and 
increase  in  on  the  job  performance  with  better  retention  of  learning 

•  Quickly  assimilate  and  convert  to  training  content  battlefield  lessons  learned 
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V  J1  AL  =  Optimize  Options  in  a  Learning!^ 
Acquisition  TRADESPACE  ^ 


The  Accelerated  Learning  Challenge 


Current  Warfighters  are  required  to  perform  tasks  for 
which  they  may  not  be  well  trained,  when  time  is  of  the 

essence. 


Examples  of  difficult  tasks  that  must  be  quickly  mastered  in: 
Irregular  Warfare  (IW), 

Counterinsurgency  (COIN),  and 


Stability,  Security,  Transition,  Reconstruction  Operations 
(SSTRO) 

•Real-time  situational  understanding 

•Determine  the  military  implications  of  fused  intelligence 
indicators,  all  source  information,  orders  of  battle  in  the 
context  of  Diplomatic,  Informational,  Military,  and/or 
Economic  (DIME)  /  Political,  Military,  Economic,  Social, 
Infrastructure  and  Information  Systems  (PMESII) 

>  Develop  options  in  Air  Operations  Centers  in  context  of 
“whole-of-government”  engagement 
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The  Accelerated  Learning  Challenge 


Examples  of  difficult  tasks  (continued) 

•Dynamic  planning/replanning 

*  Kinetic  battlespace skills 

>Maritime  battlespace  management  and  the  prevention  of 
mutual  interference  in  coalition  ops 

•  Non-kinetic  knowledge  development  and  analytic  skills 

>ln  real-time  Brigade  ops  planning  --  evaluate,  assimilate, 
and  act  in  both  the  physical  and  civil  (political,  cultural, 
and  economic)  environments  of  the  battle  space  leveraging 
non-military  organizations 

•Interpersonal  skills  (the  cultural  chameleon) 

•Achieving  and  making  use  of  societal  and  cultural  awareness 


Key  AL  S&T  Challenges  that  are 
Minimally  or  Not  Funded 


•  Understanding  the  requirements  for  proficiency  and 
how  to  accelerate  its  attainment 

•  Understanding  how  to  increase  retention  of 
competence  -  “accelerated  retention” 

•  Making  practical  serious  games  for  use  in  AL 

•  Developing  technology  for  rapid  understanding  of 
learning  problem  e.g.,  rapid  cognitive  task  analysis 

•  Determining  level  of  competence  required  for  different 
stages  of  a  war  fighter's  career 

•  Calculating  the  cost-benefit  of  implementing  AL 
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Accelerated  Learning  Principles 


Routine  practice  is  not  sufficient  for 
acceleration  of  competence.  There  needs  to 
be: 


-  A  constant  “stretching”  of  the  skill,  defined  by 
increasing  challenges  (tough  or  rare  cases) 

-  High  levels  of  intrinsic  motivation  to  work  on  hard 
problems 

-  Practice  that  provides  rich,  meaningful  feedback 

-  Practice  based  on  mentoring 

-  Provisions  for  individualized/tailored  practice  due 
to  unique  learning  styles  of  each  learner 


v  J  Example  Accelerated  Learning 

Techniques  from  Cognitive  Scienc^p^ 

Optimal  spacing  of  materials 

Spaced  is  better  than  massed  practice  (Kornell  &  Bjork,  2008;  Dempster,  1989; 
Melton,  1970;  Rohrer  &  Taylor,  2006) 

Optimal  spacing:  present  enough  repetitions  so  that  material  is  mastered,  but 
overtraining  is  an  inefficient  use  of  time 

Optimal  spacing  depends  on  how  long  material  should  be  retained  for  -  optimal 
ISI  interval  =  10-20%  Rl  interval  (Rohrer  &  Pashler,  2007) 


Using  tests  to  improve  learning,  not  just  assessment 

Advantage  of  study-test  over  study-study  (McDaniel,  Roediger,  &  McDermott, 
2007) 

The  importance  of  active  retrieval  for  learning  (Karpicke  &  Roediger,  2008) 


The  benefits  of  self-explanation  (Chi,  1996,  2000) 


Directed  comparison  for  revealing  deep  principles  (Gentner&  Namy,  2004; 
Loewenstein,  Thompson,  &  Gentner,  1999;  Rittle-Johnson  &  Star,  2007) 
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^  ^  AL  methods  for  IW/COIN/SSTRO  Readiness 


W 


Training  and  for  Mid-Career  Competence  Learnin 


Sample  Training  Guidelines  for  Developing  Proficiency 


Competent  performers  know  a  lot. 

Their  knowledge  is  highly  contextual. 

T raining  must  provide  increasingly  detailed 
knowledge,  procedures,  principles,  in  context,  with 
progressive  refinement  as  expertise  develops. 

Competent  performers’ knowledge  is 
structured. 

Provide  suitable  knowledge  structures  early  in 
training. 

Competent  performers  knowledge  /skill  is 
compiled  and  proceduralized 

Provide  sufficient  practice  for  experience  to  be 
compiled. 

Competent  performers  can  work  forward 
from  underlying  principles  or  backwards 
fromtheendgoal.lt  depends  upon  the 
case. 

Provide  underlying  principles  as  part  of  the 
knowledge  structures.  Provide  unstructured  end- 
goal  exercises  only  after  principles  have  been 
learned. 

Competent  performers  can  examine  a 
broad  range  of  alternatives  orexplorea 
single  alternative  deeply. 

Practice  environment  must  provide  for  many 
alternatives  and  must  model  them  correctly. 

Competence  keeps  developing  even  after 
many  years  and  thousands  of  opportunities 
for  deliberate  practice. 

Provide  journeyman-expert  practice  environments 
through  simulation  and  carefully  designed 
exercises. 

w  --  an  Example  of  Accelerating  Competence  -- 

Adaptive  Thinking  Training  “Think  Like  a  Commander^P 


Thinking  that  supports: 

making  adjustments  in  an  unfolding  plan  under  the 
dynamic  conditions  of  military  operations. 

Specific  Training  Task 

The  ability  to  rapidly  and  accurately  ‘size  up’  tactical 
situations:  identify  important  considerations,  risks,  key 
information,  alternative  actions,  potential  higher-order 
effects,  and  other  factors  that  may  be  significant  in 
decision  making  during  execution. 
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First  Generation  inAL 

--  an  Example  of  Accelerating  Competence  -- 


Adaptive  Thinking:  A  key  battlefield  thinking  skill — making  adjustments  in  an 
unfolding  plan  under  dynamic  conditions  of  military  operations. 


Activities  Specifically  Designed  to 
Improve  Performance 

•  Identify  desired  elements  for  expert  form 

•  Learner  performs  while  attending  to  element 

•  Coach  notes  discrepancies  from  expert  form 

•  Behavior  is  repeated  until  habitual 

•  Performance  without  attending  to  expert 
form 


Expert  Cognitive  Behaviors 

•  Keep  a  focus  on  the  mission  and  higher  intent 

•  Model  a  thinking  enemy 

•  Consider  effects  of  terrain 

•  Use  all  assets  available 

•  Consider  timing 

•  See  the  big  picture 

•  Visualize  the  battlefield 

•  Consider  contingencies  and  remain  flexible 


w  Captains  in  Command 


1.  Students  receive 

training  on  the 

expert  Themes  of 

Battlefield 

Thinkingand 

observemodel 

behavior. 


2.  Students  view  a 
3-5  minute  theme- 
based  vignette  that 
presents  a 
complex  and 
rapidly  changing 
tactical  situations. 


3.  Studentsare 
asked  to  apply 
theirtactical 
knowledgeto  think 
adaptively  and  list 
their  key 
considerations. 


4.  Students  view 
a3Dcoachorlive 
instructor 
discussing  expert 
considerations 
pertaining  to 
each  of  the 
battlefield 
thinking  skills. 

5.  The  students 
are  prompted  to 
evaluate  their 
response  based 
on  expert 
considerations 
and  respond  to 
coach’s 
questions. 

6.  After  covering 
each  of  the  eight 
themes,  students 
are  provided 
feedback  to  his  or 
her  responses. 
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Characteristics  of 
Incredibly  complex  Tasks  ( ICTs  ) 


Abstract 

Physical  phenomena  or  causation  are  not  readily  visible 

Multi-variate 

Many  variables  underlie  outcomes. 

Interactive 

Changes  in  one  variable  may  affect  several  others.  Processes  are 
co-dependent. 

Continuous 

Physical  phenomena  and  their  effects  are  described 
as  values  along  continua,  rather  than  as  discrete  properties. 

Non-Linear 

Relations  among  variables  are  not  simple  straight-line  functions 

Dynamic 

The  process  of  variation  is  of  interest,  rather  than  end-state 

Simultaneous 

Systemic  variation  is  coincident  rather  than  serial. 

Conditional 

Outcomes  are  highly  dependent  on  boundary  conditions  and 
context. 

Uncertain 

Exact  values  of  underlying  variables  are  not  known  precisely  -  they 
may  be  estimates,  interpolations,  approximations 

Ambiguous 

The  same  outcome  may  arise  from  different  combinations  of  inputs. 

13 
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nplications:  Instructional  Design  for 
Incredibly  complex  Tasks 

Abstract 

Develop  visualizations  that  explain  physical  phenomena  and  causation 

Multi-variate 

Develop  simulation-based/physics-based  problem  space  in  which  effects 
of  variation  can  be  explored 

Continuous 

Do  NOT  use  discrete  or  static  cases  for  training.  Provide  practice  for 
continuous  variation. 

Non-Linear 

Explore  the  non-linearity:  provide  practice  that  concentrates  on 
inflection  points,  minima,  maxima,  zero-crossings,  asymptotes 

Dynamic 

Practice  environment  must  include  dynamic  complexity- Scenarios  must 
present  continuous  evolution. 

Interactive 

For  high-way  interactions,  systematically  hold  some  variables  constant 
while  exploring  variation.  Use  no  more  than  three-way  interactions  for 
problem  cases. 

Simultaneous 

Develop  mental  model  for  simultaneity  as  underlying  interaction,  not 
serial  causation. 

Conditional 

Provide  highly  contextualized  practice  environment  which  is  capable  of 
supporting  practice  in  high-difficulty  real-world  warfare  environments. 

Uncertainty  & 
Ambiguity 

Teach  methods  /  procedures  for  resolving  uncertainty  /  ambiguity. 

Practice  environment  must  properly  replicate  these  effects.  Develop  test 
scenarios  which  exploit  uncertainty. 
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Paradoxes 


•  Accelerating  learning  may  have  negative  effects  on 
retention  of  what  was  learned.  The  sweet  spot  must  be 
found 

•  Accelerating  learning  in  one  area  may  have  negative 
consequences  in  terms  of  generalizability  to  a 
different  situation 

•  Shorter/faster  training  may  result  in  a  lack  of 
readiness  for  task  performance 
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Conclusions 


^OTdit  !onal  methods  ot  accelerating  the  attai nment  ot 

competence  are  required,  with  a  particular  focus  on 
military  domains 

•  To  gain  a  greater  understanding  and  improve  training 
during  these  periods,  longitudinal  and  long-term 
retention  studies  are  needed 

•  Policy  issue  is  harder  than  methodological  issues 

—  Difficult  to  obtain  funding  for  a  long-term  (10-year) 
program 

—  Project  A  was  an  exception 

•  Long-term  cross  service  program  to  collect 
personnel  data 

•  Faced  with  period  threats  of  budget  cuts 
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Annex  C  Keynote  address:  Dr.  Robert  Hoffman 


* 


Accelerated  Learning  Challenges  and  Prospects 

Robert  R.  Hoffman,  Ph.D. 

Senior  Resarch  Stientist 
Institute  for  Human  and  Machie  Cognition 
Pensacola,  FL 

Presentation  at  the 
Workshop  on  Accelerated  Learning 
Sponsored  and  Hosted  by  the 
Defence  Research  and  Development  -  Canada 
Kingston,  Ontario 
February  20 1  I 
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NEED 


Domain  practitioners  who  achieve  high  levels  of  proficiency 
provide  technical  judgment  to  speed  decision-making  in 
time-critical  events. 

They  provide  resilience  to  operations  by  resolving  tough 
problems,  acting  prudently  by  judgment  rather  than  by  rule, 
and  anticipating  future  demands  with  re-planning. 

They  exercise  effective  technical  leadership  in  ambiguous 
or  complex  situations. 


E2 


CHALLENGE 


It  typically  takes  years  of  experience  for  professionals  to 
master  their  domain. 

Reasons  for  this  include  domain  complexity  and  the  need 
for  extended  and  continuing  practice  at  rare  and  difficult 
cases. 

The  challenge  of  achieving  and  maintaining  high  proficiency 
is  compounded  in  the  military  by  such  practices  as 
collateral  assignment,  redeployment  (e.g.,  rapid  skill  decay 
on  the  part  of  pilots  creates  a  need  for  expensive  re-training), 
inadequate  or  ad  hoc  mentoring,  and  the  drive  for  “just-in-time” 
training. 
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Meanings  of  Accelerated  Learning 


Rapidized  Training 

How  to  quicken  the  training  process  while  maintaining  effectiveness 
Rapidized  Knowledge  Sharing 

How  to  rapidize  the  transposition  of  lessons  learned  from  the  battlespace 
into  the  training  context 

Accelerated  Proficiency 

How  to  train  more  quickly  to  higher  levels  of  proficiency 
Facilitated  Retention 

How  to  insure  that  training  has  a  stable  and  lasting  effect 
Rapidized  Cognitive  Task  Analysis 

How  to  rapidize  the  process  of  creating  training  materials  from  expert 
knowledge 


DOMAINS 


Cultural  understanding 
PIMESil 

Fused  sensing  and  layered  intelligence 
Influence  and  information  operations 
Second  language  learning 
Power  generation  and  coordination 
Weather  forecasting, 


Software  engineering, 
STEM  fields,  cybersecurity 
Cultural  awareness 
Personnel  management 
General  command 
Piloting  (fighters,  UAVs) 
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What  might  work? 

What  would  you  need  to  know  about  the  domain 
Concerns  and  anti-goals? 


Success  Stories? 


•  Intelligent  tutoring  systems 

•  Simulation-based  training 
•Top  Gun 

•  National  Training  Center 
•Think  Like  a  Commander 
•WRTI’s  OpSim™ 


Others? 


DRDC  Toronto  CR  201 1-105 


REVIEW  RESULTS:Training 


Training  at  advanced  levels  (training  to  perform  in  dynamic  and 
complex  domains  where  tasks  are  not  fixed)  is  generally  more 
effective  if  it  involves  extensive  practice  on  realistic  examples  or 
scenarios  (e.g.,  problem-based  learning). 

Training  has  to  balance  giving  and  withholding  of  outcome  and 
process  feedback  to  achieve  optimal  learning  at  different  stages  of 
advancement. 


BUS 


REVIEW  RESULTS: Training 


Training  at  intermediate  and  advanced  levels  benefits  significantly 
from  experience  at  challenging  problems  or  cases,  the  "desirable 
difficulties."  Short-term  performance  might  suffer,  but  longer-term 
gains  will  emerge. 

Mentoring  is  valuable  and  critical  for  advanced  learning,  because  it 
provides  opportunities  to  receive  rich  process  and  outcome 
feedback  as  learners  encounter  increasingly  complex  problems. 
However,  mentoring  is  not  always  necessary  in  advanced  stages  of 
learning. 


m 
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REVIEW  RESULTS.Transfer 


Initial  learning  that  is  more  difficult  can  lead  to  greater 
flexibility  and  transfer. 


Transferring  a  skill  to  new  situations  is  often  difficult  but  can 
be  promoted  by  following  a  number  of  training  principles. 


REVIEW  RESULTS:  Retention  and  Decay 


There  is  a  beneficial  effect  of  spacing  on  learning  and  memory 
when  the  goal  is  long  retention  intervals. 

Retention  is  better  if  the  same  task  is  never  practiced  on 
successive  trials,  but  is  randomized  with  other  practices. 

Significant  decay  can  occur  even  within  relatively  short  time 
frames  (days  to  weeks)  for  any  form  of  skill  or  learned  material, 
including  that  involved  in  military  tasks. 
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REVIEW  RESULTS:  Retention  and  Decay 


The  best  predictor  of  skill  retention  following  a  hiatus  is  the 
level  of  performance  achieved  (including  over-learning)  prior  to 
the  hiatus. 

The  variable  having  the  largest  impact  on  performance  after  a 
retention  interval  is  the  similarity  of  the  conditions  of  testing  to 
those  of  the  training. 

Overlearning  is  the  prime  determinant  of  memory  and  skill 
decay.'The  greater  the  degree  of  overlearning  the  less  decay  and 
the  less  rapid  the  decline. 
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REVIEW  RESULTS 


None  of  the  above  generalizations  holds  uniformly; 
for  all  of  the  above,  one  can  find  studies  showing 
opposite  effects,  or  no  effects. 


Learning,  training  and  instructional  design  are  highly  complex, 
with  multiple  interacting  factors,  makes  it  impossible  to 
compose  a  recipe  for  instructional  design  for  proficiency 
training. 
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REVIEW  RESULTS 


In  an  ideal  situation,  one  would  be  able  to  trace  the  effects 
of  instruction,  from  the  very  beginning,  noting  its  effect  across 
the  entire  span  of  learning  various  knowledge 
and  skills,  right  through  to  final  operations  performance. 


Reasons  why  this  is  not  possible 
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INDETERMINATE  CAUSATION 


* 


The  time  taken  for  the  changes  due  to  training  to  percolate 
through  to  operational  effects  can  be  long,  and  involves 
complex  interactions  with  other  functional  realities,  such 
as  organizational  constraints  and  operational  pressures. 

There  is  no  straightforward  track,  from  the  formal  process 
of  lessons-learned  to  eventual  operational  performance, 
that  would  enable  us  to  give  credit  to  any  particular  training 
factor. 
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INDETERMINATE  CAUSATION 


Operational  outcomes  are  so  contextually  and  circumstantially 
driven  that  they  defy  backwards-looking  interpretation 
necessary  to  understand  cause-and-effect  linkages. 


Operational  effectiveness  outcomes  are  tied  to  scenarios 
and  circumstances,  making  it  impossible  to  get  representative 
coverage  permitting  generalizations  that  specify  which  training 
inputs  are  effective  in  meeting  which  possible  or  future 
operational  challenges. 
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THE  PARADOX  OF  JOBS 


As  technology  changes,  as  assignments  change, 
and  as  jobs  change,  the  cognitive  work  changes. 

Learning  and  re-learning  on  the  job  must  be  continuous. 

But  there  is  too  much  to  train,  in  too  little  time. 

Thus,  the  pressure  is  to  do  Just-in-Time  training, 

but  such  training  is  at  odds  with  notions  of  proficiency 

training. 

The  practice  of  rotating  the  most  highly-trained  and 
experienced  pilots  to  other  duties  and  responsibilities 
and  then  sometimes  rotating  them  back  into  piloting 
or  pilot  training. 

20 


THE  PARADOX  ofTASKS 


Traditionally,  training  depends  on  componential  analysis 
of  relatively  stable  tasks  that  can  be  described  as  series 
of  steps  or  procedures. 

But  cognitive  work  is  the  context-sensitive,  knowledge-driven 
choice  among  alternative  activities. 

The  traditional  concept  of  “task”  may  actually  be  an 
impediment  to  advances  in  research  and  theory  in  that 
it  reinforces  an  artificial  notion  of  separability. 
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PARADOXES  of  CAREERS 


* 


Paradox  of  Domain  Specificity 
The  Envisioned  World  Paradox 
The  Moving  Target  Paradox 
Paradox  of  Feedback 
The  Skills-Capacities  Paradox 
Paradox  of  Tough  Tasks 
The  Paradox  of  Failure 
The  Irony  of  Evaluation 
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Summary 


Can  learning  be  accelerated  (how  much)? 

For  fixed  tasks,  yes. 

For  dynamic  tasks,  likely 
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Summary 


Does  acceleration  of  learning  penalize  retention? 

We  do  not  know. 

Question  is  not  does  it  penalize,  but  how  does  it 
impact? 

How  does  acceleration  of  learning  retention  following 
hiatus? 

We  do  not  know. 
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Research  Needs:  Methodology 


Research  must  establish  methods  for  rapidized  cognitive  task 
analysis. 

The  most  frequently  mentioned  training  challenge  was  the  pace 
of  change  in  domains  of  professions  practice,  including  in 
military  jobs  and  specialties. 
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Research  Needs:  Methodology 

What  makes  for  a  good  mentor? 

How  to  identify  individuals  who  could  become  good  mentors? 

How  to  teach  people  how  to  be  mentors? 

What  are  the  most  efficient  and  appropriate  ways  of  using 
mentoring  to  achieve  accelerated  learning? 


26 


m 


Research  Needs:  Methodology 


Expand  our  empirical  base  about  how  periodic  booster  sessions 
(number,  spacing)  might  affect  forgetting  rates. 

Expandour  empirical  base  about  differential  decay  of  particular 
abilities. 

Booster  training  should  focus  on  refresh  of  skills  that  would 
ordinarily  show  faster  drop-off. 

Procedures  for  rapidized  updating  both  during  and  post-hiatus, 
because  during  a  hiatus,  the  missions  and  the  technology  are 
constantly  changing. 
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Acceleration  Methods 


Computer  games,  simulations,  and  immersions,  all  of  which  are 
enhanced  through  application  of  intelligent  tutoring  and  artificial 
intelligence  capabilities. 

Case-based  instruction  and  realistic  tough  cases  with  focus  on 
errors  and  "desirable  difficulties." 

Meaningful  corrective  feedback  that  is  appropriately  timed  (neither 
too  close  nor  too  distance  from  the  performance  being  evaluated). 

Tough-case  time  compression 


Summary  Roadmap 


Where  should  R&D  (in  behavioural  sciences  or  technology)  be 
expended  to  best  effect? 


S  >  2  pilot  projects,  interestingly  different  domains 
^  Longitudinal  (cohort-select) 

S  Domans  with  civilian  analogs 

S  Domains  with  recently  transformed  strategic  environment, 
where  people  have  more  than  one  duty  assignment 

S  Dynamic  tasks  and  challenges  (versus  "tasks") 

S  Intyegrated  knowledge  management  component 

29 
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Summary  Roadmap 


Prospective  and  retrospective  studies. 

The  paramount  goals  of  a  research  program  would  be: 

( 1 )  Facilitating  the  achievement  of  proficiency,  especially  in  the 
Apprentice  to  Senior  Journeyman  levels, 

(2)  Retaining  expertise  in  the  form  of  both  personnel 
capabilities  and  in  the  form  of  organizational  knowledge, 

(3)  Applications  to  military  domains  including  mission-critical 
specializations. 


www.ihmc.us 


www.ihmc.us/users/rhoffman/main 
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Annex  D  Presentation  of  topics 


Accelerated  Learning  and 
Retention  Workshop  -  n  Feb  2011 

Dr.  Lochlan  Magee  -  DRDC  Toronto 

LCdr  Peter  Ball.  Lt(N)  Calixte  Wondje 
and  Capt.  Gregory  Jones  -  CDA 

Lisa  Rehak,  Cheryl  Karthaus  and  Barb  Adams  - 
Humansystems 

©  Human  systems1  Incorporated 
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*  Quick  overview  of  literature  review 

*  Identification  of  emergent  topics 

*  Definition,  description  of  each  topic 

*  Examples  of  questions  that  might 
emerge 


©  Humansystems*lncorporated 


Literature  Review 
Process 


Keywords 

*  Accelerated  learning 

*  Skill  and  knowledge 
retention 

Reviewed 

*  Academic 

*  Industry 

*  Public  media  sources 

Total  of  26  articles 
were  reviewed 
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of  the  Scan 

*  Identified  a  wide  range  of  factors  that 
affect  learning  and  retention  of 
knowledge  and  skills 

*  Many  gaps  in  relevant  literature 

*  Narrowed  to  is  key  topics  related  to 
learning  (12)  and  retention  (3) 


Two  Theme  Categories 
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Learning  Theme  Categories 


Learning  Theme  Categories 
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Learning  Theme  Categories 


i)  Student  attributes 

*  Students  naturally  vary  in  their 
learning  potential  and  style 

•  Prominent  dimensions  are  cognitive 
ability,  motivation,  learning  styles, 
prior  experience  (e.g.,  training  and 
education) 
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I  Student  Attributes: 
Example  Questions 

•  What  student  attributes  are  most 
critical  to  learning? 

•  How  can  training  be  designed  to 
maximize  student  motivation? 

•  Should  we  further  investigate  the 
importance  of  learning  style 
preference? 

©  Humansysfems'lncorporated 


Learning  Theme  Categories 
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Instructor  Attributes 

*  Level  of  subject  matter  expertise 

*  Teaching  experience 

*  Teaching  style 


Instructor:  Example  Questions 

•  What  are  the  most  critical  aspects  of 
instructor  effectiveness? 

•  What  is  the  ideal  mix  of  subject 
matter  expertise  and  teaching 
experience? 

•  What  makes  a  good  mentor? 
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Learning  Theme  Categories 


Media 

3)  Classroom  aids  and  environment 

4)  Distance  learning 

5)  Simulation-based  instruction 

6)  Computer-based  instruction 

7)  Embedded  training 
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3)  Classroom 
aids/environment 

*  Aids  -  eg,  concept  maps,  videos 

*  Environment  -  noise,  lighting,  music 


•  Example  Question: 

•  What  aids  are  most  conducive  to 
learning? 
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4)  Distance  learning 

*  Defined  as  learning  without  physical 
co-location  of  information  and 
learners 

*  Examples  include  e-learning,  m- 
learning  (learning  with  mobile 
devices  such  as  iPads) 


©  Human  systems*  Incorporated 
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Distance  learning:  Example 
Questions 

*  What  skills  are  most  conducive  to 
the  use  of  distance  education? 

*  Do  skills  taught  via  distance 
education  show  similar  rates  of 
decay? 
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p)  Computer-based 
'  instruction 

*  Using  computers  as  part  of  training 
and  education  delivery 

*  Varying  levels  of  complexity, 
ranging  from  low  (e  g.,  typing  tutor) 
to  intelligent  tutoring  systems 

*  Animation  and  interactive 
microworlds 

©  Humansysfems'lncorporated 
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[Computer-based  instruction: 
1  Example  Questions 


*  Can  computer-based  training 
accelerate  learning? 

*  Can  computer-based  training 
facilitate  better  levels  of  transfer  to 
work  performance? 


©  Humansysfems'lncorporated 

ITT  Simulator-based 
'  instruction 

*  Simulation  -  working  representation 
of  reality  used  to  represent  devices 
and  processes  -  provides  cues  to 
trigger  learner  responses 

0  All  kinds  of  human-in-the-loop  (HBR) 
and  VR-based  instruction 

0  Ranges  from  low  to  high  fidelity 
virtual  environments 

©  Humansysfems*lncorporated 
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[Simulator-based  instruction: 
Example  Questions 

*  How  much  fidelity  is  enough? 

*  What  tradeoffs  can  be  made 
between  simulation-based 
instruction  and  actual  flight  time  / 
range  time/  sea  time? 

*  In  team  environments,  can  SBI 
models  substitute  for  team 
members? 

©  Humansystems*lncorporated 


5)  Embedded  training  (ET) 

*  Training  that  is  built  into  or  added  to 
a  weapon  system 

-Witmer  &  Knerr,  1996 

*  Embedded,  appended,  or  umbilical 

*  Allows  for  continual  training 


©  Humansysfems'lncorporated 
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ET:  Example  Questions 

•  What  is  the  most  effective  process  to 
design,  develop,  and  test  ET 
solutions? 

•  How  can  new  technology  such  as 
fused  reality  be  used  to  enhance 
performance? 


©  Humansysfems'lncorporated 

Learning  Theme  Categories 
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Methods 

8]  Collaborative  learning 

9]  Scheduling 

10]  Feedback  and  testing 

11]  Adaptive  instruction 

12]  Context  and  content 


8)  Collaborative  learning 

*  Approaches  that  require  joint 
intellectual  efforts  by  one  or  more 
students 

*  E.g.  -  social  networking  tools  such  as 
Facebook,  Twitter  etc. 


•  Example  Question: 

*  Do  collaborative  tools  promote 
learning? 

©  Humansysfems'lncorporated 
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p)  Scheduling 


*  When  and  how  should  practice  and 
testing  occur 

*  Timing,  duration  and  spacing  (e  g., 
massed  vs.  distributed)  of  training 
and  practice 

*  Different  ways  to  schedule  training 
(e.g.,  just-in-time  training,  time 
compressed) 

©  Humansysfems'lncorporated 


:  Example 

•  How  should  practice  be  scheduled 
(spaced  or  massed)  for  varying 
types  of  skills? 

•  What  is  the  relative  effectiveness  of 
different  training  schedules? 


[Scheduling 
1  Questions 


©  Humansysfems'lncorporated 
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io)  Feedback  and  testing 

*  What  type  of  feedback  (outcome, 
process,  intrinsic)? 

*  Frequency  of  feedback  (continuous, 
intermittent 

*  Timing  -  immediate  vs.  delayed 

*  Source  -  peers,  instructor,  system 


Feedback  and  testing: 
Example  Questions 

*  What  are  the  optimal  rates  of 
feedback? 

*  What  are  the  optimal  rates  of 
testing? 
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ii)  Adaptive  instruction 

*  Tailoring  of  training  and  education 
to  the  needs  of  the  individual  (e  g., 
intelligent  tutoring] 

*  More  interesting  for  students  and  as 
well  as  increasing  their  motivation 
and  consequently  their  learning 

*  Use  of  neuroscience  to  guide 
adaptation 


r/Vdaptive  instruction: 
1  Example  Questions 


*  What  types  of  adaptive  instruction 
are  most  critical  to  promoting 
accelerated  learning  (e  g.,  rate  of 
presentation,  practice?) 

•  How  can  neuroscience  best  be 
applied  to  military  teams? 
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12)  Context  and  Content 

*  Teaching  why  vs.  how  or  what 

*  Environment  -  stress,  cognitive 
workload,  difficulty 

*  Content  -  cross-training,  mixing 
team  members,  overlearning, 
"tough"  cases 


[Context  and  Content: 
Example  Questions 

*  Can  overlearning  help  to  accelerate 
learning? 

*  How  much  stress  is  best? 

*  Is  cross-training  a  good  method  of 
promoting  learning  and  retention? 
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Two  Theme  Categories 


Retention  Theme  Categories 


Models  of  Techniques  for 
Retention  Retention 


Learning 

Factors 


DRDC  Toronto  CR  201 1-105 


143 


Learning 

Factors 


Techniques  for 
Retention 


13)  Relationships  to  Learning 
Factors 

•  Maintenance  of  learned  behaviours 
without  practice 

*  Implications  of  skill/knowledge 
acquisition  for  retention 


©  Human  systems*  Incorporated 
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[Relationships  to  Learning 
1  Factors:  Example  Questions 

*  Does  acceleration  of  learning  impact 
negatively  on  retention? 
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Retention  Theme  Categories 
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14)  Models  of 

knowledge/skill  retention 

*  Skills  decay  at  different  rates 

*  How  long  will  skills/knowledge  last? 

*  When  is  refresher  training 
necessary? 


©  Humansysfems'lncorporated 

[Models  of  knowledge/skill 
1  retention:  Example  Questions 

*  Can  models  of  skill  retention  predict 
skill  decay? 


©  Human  systems*  Incorporated 
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is)  Techniques  for  Retention 

•  Techniques  intended  to  develop  or 
aid  memory 

*  E  g.,  mnemonics  (acronyms,  poems) 


Example  Question: 

*  What  retention  techniques  are  the 
most  effective?  (e.g.,  mental 
rehearsal) 


©  Hu mansystems* Incorporated 
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of  Learning  Topics 

1]  Student  attributes 

2)  Instructor  attributes 
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[Summary  of  Learning  Topics 
'(continued) 


Delivery  Media 

3)  Classroom  aids/environment 

4)  Distance  learning 
s)  Embedded  training 

6)  Computer-based  instruction 

7)  Simulator-based  instruction 
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[Summary  of  Learning  Topics 
'(continued) 


Delivery  Methods 

8)  Collaborative  learning 

9)  Scheduling 

10)  Feedback  and  testing 

1 1 )  Adaptive  instruction 

12)  Context  and  Content 
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of  Retention 

13)  Relationships  to  learning  factors 

14)  Models  of  knowledge/skill 
retention 

is)  Techniques  for  Retention 


[Summary 

'Topics 


©  Human  systems*  Incorporated 
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I5 


Conclusion 


•  Fifteen  (is)  topics  emerged  related  to 
learning  and  retention 

•  Workshop  discussions  will  further 
elaborate  on  the  value  of  these 
topics 


©  Humansys(ems*lncorporated 
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Annex  E  Rating  Matrix  -  Topic  by  Criteria  Ratings 
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Topics 

A.  -  The  topic  has 
scientific  merit. 

B.  -  The  topic  is 
novel  or  innovative 
in  addressing 
learning  and 
retention. 

C.  -  The  topic 
aligns  with  and 
supports  CF/DND 
goals. 

D.-The  topic  will 
have  wide 
application  within 
CF/DND. 

E.  —  The  topic  will 
improve 
organizational 
efficiency  and/or 
effectiveness. 

F.  -  The  topic  will 
address  identified 
gaps  -  learning  and 
retention  deficits  - 
in  the  CF/DND. 
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Defence  Research  &  Development  Canada 

Director  Research  and  Development  Knowledge  and  Information 
Management 

Research  &  Development 
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aux  objectifs  des  FC  ou  du  MDN  et  le  potentiel  de  collaboration  au  niveau  national  ou 
international. 
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